Foodstuff sensing appliance

ABSTRACT

An appliance includes a temperature control system, a light source, and a spectrometer. The spectrometer works in conjunction with the light source to obtain a measured value of a foodstuff. The measured value is at least one of an absorbance value, a transmittance value, and a reflectance value. The measured value from the foodstuff is compared to a reference value for the foodstuff.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 62/201,328, filed onAug. 5, 2015, entitled “OBJECT RECOGNITION SYSTEM FOR AN APPLIANCE ANDMETHOD FOR MANAGING HOUSEHOLD INVENTORY OF CONSUMABLES,” the entiredisclosure of which is hereby incorporated herein by reference. Thisapplication also claims priority to and the benefit under 35 U.S.C. §120 of U.S. Non-provisional patent application Ser. No. 15/198,258,filed on Jun. 30, 2016, which stems from the above-noted provisionalapplication and is entitled “OBJECT RECOGNITION SYSTEM FOR AN APPLIANCEAND METHOD FOR MANAGING HOUSEHOLD INVENTORY OF CONSUMABLES,” the entiredisclosure of which is hereby incorporated herein by reference.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to an appliance. Morespecifically, the present disclosure relates to an appliance that isconfigured to sense a foodstuff.

BACKGROUND

This application relates to household appliances, and more typically arefrigerating appliance or cooking appliance. In particular, theappliance disclosed herein incorporates a system that capturessuccessive inputs of data and analyzes and compares the captured data orportion of the data to provide an indication to the user of theappliance about the status of the household inventory of variousconsumable products.

Appliances are often provided with a temperature control system.Additionally, some appliances are provided with storage compartments,such as refrigerators. The storage compartments are often utilized forstoring foodstuffs. However, due to the large storage capacity of theappliance, such as the refrigerator, users can have a difficult timeusing the foodstuffs prior to the spoilage of the foodstuffs.Accordingly, there is a need for appliance solutions that aid the userin keeping track of the shelf life and/or spoilage of the foodstuffs.

SUMMARY OF THE DISCLOSURE

In at least one aspect of the disclosure, a refrigerating applianceincludes a cabinet having an inner liner that defines at least onecabinet aperture for accessing a corresponding refrigeratingcompartment, at least one door operably coupled to the cabinet andoperable between open and closed positions, the open position defined bythe refrigerating compartment being accessible through at least onecabinet aperture, and the closed position being defined by the doorengaging at least a portion of at least one aperture. A plurality ofstorage surfaces is positioned within the refrigerating compartment,each of the plurality storage surfaces adapted to receive at least onetemporarily stored item. One or more scanning apparatus is adapted torecord data related to the plurality of storage surfaces and at leastone temporarily stored item. One or more scanning apparatus conductsfirst and second data capturing phases. The secondary data is recordedduring the second data capturing phase is analyzed and compared againstprevious data recorded during the first data capturing phase to create acurrent inventory manifest. The current inventory manifest is comparedwith a set of predetermined parameters related to the at least onetemporarily stored item to selectively provide an output.

In at least another aspect, an appliance includes a cabinet having aninterior compartment, wherein the cabinet defines at least onecorresponding aperture for accessing the interior compartment. At leastone door panel is coupled to the cabinet and operable between open andclosed positions. A storage area is defined within the interiorcompartment, the storage area adapted to receive at least onetemporarily stored item. A scanning apparatus is adapted to record datarelated to the storage area and the at least one temporarily storeditem. The scanning apparatus conducts first and second data capturingphases, and secondary data recorded during the second data capturingphase is compared against previous data recorded during the first datacapturing phase to create a current inventory manifest. The currentinventory manifest is compared with a set of predetermined parametersrelated to the at least one temporarily stored item to selectivelyprovide an output.

In at least another aspect, an object recognition system for anappliance includes a plurality of storage surfaces positioned within astorage area, each of the plurality storage surfaces adapted to receiveat least one temporarily stored item. A scanning apparatus is adapted torecord data related to the plurality of storage surfaces and the atleast one temporarily stored item, wherein the scanning apparatusconducts first and second data capturing phases. Secondary data isrecorded during the second data capturing phase is compared againstprevious data recorded during the first data capturing phase to create acurrent inventory manifest. The current inventory manifest is comparedwith a set of predetermined parameters related to the at least onetemporarily stored item to selectively provide an output related to atleast one temporarily stored item.

In at least one aspect, an appliance includes a temperature controlsystem, a light source, and a spectrometer. The spectrometer works inconjunction with the light source to obtain a measured value of afoodstuff. The measured value is at least one of an absorbance value, atransmittance value, and a reflectance value. The measured value fromthe foodstuff is compared to a reference value for the foodstuff.

In at least another aspect, an appliance includes a temperature controlsystem, a light source, a controller, one or more monitored areas, and agas detector. The gas detector is coupled, and in communication, with atleast one of an imager and a spectrometer such that the one or moremonitored areas are monitored to identify a first state of a foodstuff.The controller compares the first state of the foodstuff with a secondstate of the foodstuff.

In at least another aspect, a method of assembling an appliance includesthe steps of coupling a temperature control system to the appliance,coupling a gas detector to the appliance, coupling an imager to theappliance, coupling at least one of the gas detector and the imager to acontroller, configuring the gas detector to monitor an area of theappliance, and configuring the controller to collect data from at leastone of the gas detector and the imager.

These and other features, advantages, and objects of the present devicewill be further understood and appreciated by those skilled in the artupon studying the following specification, claims, and appendeddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofthe device, will be better understood when read in conjunction with theappended drawings. For the purpose of illustrating the device, there areshown in the drawings, certain embodiment(s) which are presentlypreferred. It should be understood, however, that the device is notlimited to the precise arrangements and instrumentalities shown.Drawings are not necessary to scale. Certain features of the device maybe exaggerated in scale or shown in schematic form in the interest ofclarity and conciseness.

FIG. 1 is a front perspective view of a French door bottom mount (FDBM)refrigerator with the refrigerator compartment doors open andillustrating various exemplary aspects of a scanning apparatus for theobject recognition system;

FIG. 2 is a front elevational view of a refrigerating applianceincorporating various exemplary external aspects of a scanning apparatusfor the object recognition system;

FIG. 3 is a perspective view of an interior of an exemplaryrefrigerating appliance incorporating various temporarily stored itemsin a first position during a first data capturing phase;

FIG. 4 is a perspective view of the exemplary appliance of FIG. 3 withthe temporarily stored items in a second position during a second datacapturing phase;

FIG. 5 is a cross-sectional view of the refrigerating appliance of FIG.2 illustrating an exemplary data scanning phase of temporarily storeditems within a refrigerating compartment;

FIG. 6 is a lateral cross-sectional view of an exemplary refrigeratingappliance with the doors in the open position and the scanning apparatusperforming a data scanning phase of the interior surfaces of theopposing French doors;

FIG. 7 is a series of cross-sectional views of an exemplary applianceillustrating the process of a data scanning phase as the doors are movedfrom a closed position to an open position in order to capture athree-dimensional image of the refrigerating compartment and thetemporarily stored items;

FIG. 8 is a linear flow diagram illustrating an exemplary process forcapturing a current inventory manifest for the object recognitionsystem;

FIG. 9 is a flow diagram illustrating an exemplary process forformulating a shopping list for the object recognition system;

FIG. 10 is a flow diagram illustrating an exemplary process forformulating a shopping plan utilizing the object recognition system;

FIG. 11 is a flow diagram illustrating an exemplary process forformulating a shopping list utilizing an aspect of the objectrecognition system;

FIG. 12 is a flow diagram illustrating an exemplary process for managinginventory of the current inventory manifest utilizing an aspect of theobject recognition system;

FIG. 13 is a schematic diagram illustrating an aspect of the objectrecognition system for assessing spoilage of various temporarily storeditems;

FIG. 14 is a schematic diagram illustrating an aspect of the objectrecognition system for managing the current inventory manifest inconjunction with one or more appliances to determine object placement;

FIG. 15 is a schematic diagram illustrating an aspect of the objectrecognition system for inputting temporarily stored items within acurrent inventory manifest, according to at least one embodiment;

FIG. 16 is a schematic diagram illustrating an aspect of the objectrecognition system for providing communication between an applianceutilizing the object recognition system and a personal computing devicethat can communicate with information gathered by the object recognitionsystem;

FIG. 17 is a front perspective view of an appliance incorporating anaspect of the scanning apparatus and illustrating an exemplary aspect ofthe scanning apparatus;

FIG. 18 is a front perspective view of an appliance incorporating anaspect of the scanning apparatus and illustrating an exemplary aspect ofthe scanning apparatus and further illustrating a U-shaped tuck shelfsystem in an untucked configuration;

FIG. 19 is a front perspective view of an appliance incorporating anaspect of the scanning apparatus and illustrating an exemplary aspect ofthe scanning apparatus and further illustrating a U-shaped tuck shelfsystem in a tucked configuration;

FIG. 20 is a front perspective view of a FDBM refrigerator with therefrigerator compartment doors open and bottom freezer compartment doorclosed;

FIG. 21 is a perspective view of the U-shaped shelves of a FDBMrefrigerator, showing temporarily stored items positioned on the shelveswithin the refrigerator cabinet;

FIG. 22 is a perspective view of two piece U-shaped shelves within aFDBM refrigerator;

FIG. 23 is a perspective view of two piece U-shaped shelves within aFDBM refrigerator where the shelves have glass inserts which may beremovable shelves or tuck shelves;

FIG. 24 is a front view of an appliance having access doors in an openposition, according to one embodiment;

FIG. 25A is a front perspective view of a storage compartment having alarge quantity of a foodstuff stored therein, according to oneembodiment;

FIG. 25B is a front perspective view of the storage compartment having asmall quantity of the foodstuff stored therein, according to oneembodiment;

FIG. 26A is a front perspective view of the storage compartment havingthe large quantity of the foodstuff stored therein, according to anotherembodiment;

FIG. 26B is a front perspective view of the storage compartment havingthe small quantity of the foodstuff stored therein, according to anotherembodiment;

FIG. 27A is a front perspective view of the storage compartment havingthe large quantity of the foodstuff stored therein, according to afurther embodiment;

FIG. 27B is a front perspective view of the storage compartment havingthe small quantity of the foodstuff stored therein, according to afurther embodiment;

FIG. 28 is a front perspective view of the storage compartment having avariety of foodstuffs stored therein, according to yet anotherembodiment;

FIG. 29A is a plot depicting exemplary data collected from one or moresensors that are coupled to the appliance and may be utilized in apredictive manner;

FIG. 29B is a plot depicting exemplary data collected from the one ormore sensors that are coupled to the appliance;

FIG. 30 is a side perspective view of the appliance, according toanother embodiment, having a plurality of sensors coupled thereto;

FIG. 31 is a side perspective view of the appliance, according toanother embodiment, configured to cook the foodstuff;

FIG. 32 is a schematic representation of the appliance, according to oneembodiment;

FIG. 33 is a flow chart depicting a method of operating the appliance,according to one embodiment;

FIG. 34 is a flow chart depicting a method of assembling the appliance,according to one embodiment; and

FIG. 35 is a flow chart depicting a method of operating an appliance,according to one embodiment.

DETAILED DESCRIPTION

Before the subject device is described further, it is to be understoodthat the device is not limited to the particular embodiments of thedevice described below, as variations of the particular embodiments maybe made and still fall within the scope of the appended claims. It isalso to be understood that the terminology employed is for the purposeof describing particular embodiments or aspects of embodiments, and isnot intended to be limiting. Instead, the scope of the present devicewill be established by the appended claims.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimit of that range, and any other stated or intervening value in thatstated range, is encompassed within the device. The upper and lowerlimits of these smaller ranges may independently be included in thesmaller ranges, and are also encompassed within the device, subject toany specifically excluded limit in the stated range. Where the statedrange includes one or both of the limits, ranges excluding either orboth of those included limits are also included in the device.

In this specification and the appended claims, the singular forms “a,”“an” and “the” include plural reference unless the context clearlydictates otherwise.

Referring to an aspect of the various embodiments illustrated in FIGS. 1and 2, reference numeral 510 generally refers to a household appliance,typically a refrigerating appliance. The appliance 510 can include acabinet 512 having an inner liner 514 that defines at least one aperture516 for accessing a corresponding refrigerating compartment 518. Therefrigerating compartment 518 can be a refrigerated compartment 520,freezer compartment 522, warmer, crisper, pantry compartment 524,combinations thereof, and other similar compartments found within arefrigerating appliance 510. At least one door 526 is operably coupledto the cabinet 512 and is operable between open and closed positions528, 530. It is contemplated that the open position 528 is defined bythe refrigerating compartment 518 being accessible through the at leastone aperture 516. In the case of a French door refrigerator, the openposition 528 can be defined by one of the French doors 526 being openand one of the French doors 526 being closed. The closed position 530 isdefined by the door 526 engaging at least a portion of the at least oneaperture 516, such that either the refrigerating compartment 518 isinaccessible or only partially accessible. While discussed and describedin the context of a French door bottom mount (FDBM) configurationappliance, the present disclosure may be applied to other configurationsincluding a dual pantry drawer, side-by-side, top mount freezer, andfour-door configuration, for example.

Referring now to FIGS. 1-8, a plurality of storage surfaces 540 can bepositioned within one or a plurality of refrigerating compartments 518,where each of the plurality of storage surfaces 540 within therefrigerating compartment 518 is adapted to receive at least onetemporarily stored item 542, which is typically a food or beveragecontainer or a food item. Examples of such temporarily stored items 542can be food items, condiments, produce, meats, frozen items, spices,baking needs, beverages and other similar consumable or perishable itemsgenerally stored within a household appliance 510. The appliance 510 caninclude a scanning apparatus 544 that is adapted to record data relatedto the plurality of storage surfaces 540 and the various temporarilystored item or items 542 placed thereon. In this manner, the scanningapparatus 544 is configured to conduct various data capturing phases550, including first and second data capturing phases 552, 554. The datacapturing phases 550 can be performed to record, obtain, scan, read,measure, view, input, transfer, sequester or otherwise capture datathrough the use of the scanning apparatus 544. According to the variousembodiments, secondary data 556 recorded during the second datacapturing phase 554 can be analyzed and compared against previous data558 recorded during the first data capturing phase 550 to create acurrent inventory manifest 560. This current inventory manifest 560 canbe compared with a set of predetermined parameters 562 related to the atleast one temporarily stored item 542. When the current inventorymanifest 560 is compared against the predetermined parameters 562 thatare related to the at least one temporarily stored item 542, a processor670 for the appliance 510 generates an output such as in the form of anindicia transmitted to a user that can correspond to any one or more ofthe at least one temporarily stored item 542. The predeterminedparameters 562 can relate to user preferences as to quantity, quality,life span, spoilage, and other information related to the varioustemporarily stored items 542 or combinations of parameters, including,but not limited to, the parameters discussed above.

It is contemplated that the processor 670 can be located within aportion of the appliance 510. It is also contemplated that the processor670 can be a “cloud” based computing system positioned remote from theappliance 510 (shown in FIG. 16).

According to the various embodiments, as exemplified in FIGS. 1-8 and17-19, the scanning apparatus 544 can include a plurality of cameras 570disposed within the refrigerating compartment 518 of the appliance 510.The cameras 570 can be configured to periodically scan each of theplurality of storage surfaces 540, as well as the at least onetemporarily stored item 542 disposed within any one or more of theplurality of storage surfaces 540. In this manner, during the first andsecond data capturing phases 552, 554, the scanning apparatus 544 canrecord a photograph, or other type of image, of the interior 572 of therefrigerating compartment 518, and/or can record the data usingalternate imaging methods that can include, but are not limited to,infrared imaging, 3-D imaging, light-field imaging, thermal imaging,digitized imaging, combinations thereof, and any other type of imagingmethod that can collect data as to the various properties andconfigurations of the plurality of storage surfaces 540 and the at leastone temporarily stored item 542 disposed within.

According to various aspects of the device, as illustrated in FIGS.1-16, it is contemplated that the current inventory manifest 560generated through data recorded during the first and second datacapturing phases 552, 554 can be implemented in numerous methods forassisting the user in maintaining the household inventory of items, bothwithin the appliance 510 and throughout various portions of the house,as will be described more fully below.

The scanning apparatus 544 can include at least one or more cameras 570.The camera(s) 570 can be adapted to record the contents of therefrigerating compartment 518, such as the at least one temporarilystored item 542 being disposed within one or more of the plurality ofstorage surfaces 540. The camera(s) 570 also can be adapted to recordthe contents of at least one of the one or more temporarily stored items542. By way of example, and not limitation, the scanning apparatus 544can record the position of a substantially transparent milk container578 (or other similar at least partially translucent, translucent ortransparent container) disposed within the refrigerating compartment518. Additionally, the scanning apparatus 544 can capture or readinformation regarding the amount of milk contained within thetranslucent milk carton. In this manner, during the first data capturingphase 552, the scanning apparatus 544 will record or read thepositioning of each of the temporarily stored items 542 as well as theremaining contents 580 contained therein.

During the second data capturing phase 554, a substantially similar scanis taken of the interior 572 of the refrigerating compartment 518 togather updated data concerning the contents of the refrigeratingcompartment 518. The data obtained through the first and second dataobtaining, more typically, data recording phases 552, 554 is compared,and the differences in the two images or data sets are analyzed todetermine changes in the location of the plurality of storage surfaces540, removal of one or more of the storage surfaces 540, changes in thelocation of one or more of the temporarily stored items 542, removal ofone or more of the temporarily stored items 542, the changes in theremaining contents 580 of one or more of the temporarily stored items542, changes in the physical appearance of one or more of thetemporarily stored items 542, combinations thereof, and other similarcomparative information that can be derived through a comparison of dataobtained during the first and second data capturing phases 552, 554. Asdiscussed above, the analysis and comparison of the first and seconddata capturing, typically recording, phases 552, 554 produces thecurrent inventory manifest 560 of the appliance 510 that corresponds tothe current configuration and contents of the refrigerating appliance510, and also changes as to the contents of the refrigerating appliance510 since performance of the first data capturing phase 552.

According to various embodiments, it is contemplated that a set of datacaptured during a particular data capturing phase 550 may not becompared with another data set. In such an embodiment, the datacapturing phase 550 can record a current state of the contents of theappliance 510.

According to the various embodiments, the scanning apparatus 544 havingcameras 570 may be limited for purposes of determining the contents oftemporarily stored items 542 that may have an opaque or only slightlytranslucent outer covering, such as an opaque plastic container 578 ofmustard, or a cardboard container 578 of milk. In such a situation, aphotographic image may not be sufficient or completely sufficient todetermine the status of the contents of the temporarily stored item 542.Accordingly, the scanning apparatus 544 can also include various scales590 or systems to weigh items disposed within portions of the pluralityof storage surfaces 540 that can measure the remaining contents 580 of aparticular container 578, based upon observed changes in the weight ofthe temporarily stored item 542.

According to the various embodiments, the scales 590 and cameras 570 canwork in conjunction, such that the cameras 570 can determine thelocation of a particular temporarily stored item 542, and the scales590, where the container 578 is substantially opaque, can determine thecontents remaining within the container 578 for the temporarily storeditem 542. Other imaging systems that may be able to obtain the amount ofa food or beverage in an opaque container such as a thermal imagingcamera or ultrasound system(s) can also be incorporated into thescanning apparatus 544.

According to various aspects of the device, as illustrated in FIGS. 3-8,it is contemplated that the first and second data capturing or recordingphases 552, 554 can be undertaken at various times during the use of theappliance 510. According to at least one aspect, when the door of theappliance 510 is moved to a closed position 530, the scanning apparatus544 can be engaged to conduct a data capturing phase, typically arecording phase 550. In this manner, after a user of the appliance 510is done viewing, removing, using, disposing, replacing and/or otherwisemanipulating the various temporarily stored items 542, such as duringpreparation of meals, the user will close the door and the scanningapparatus 544 conducts the data capturing phase 550. In this manner, thefirst data capturing phase 552 is conducted during a movement of thedoor 526, and the second data capturing phase 554 is conducted during asubsequent movement of the door 526. Accordingly, after the user is donemanipulating the various temporarily stored items 542 within theappliance 510, the data capturing phase 550 can be undertaken and thevarious analyses and comparative computations can be made, typically byone or more computer or server systems, to derive the current inventorymanifest 560 of the appliance 510. It is also contemplated that thescanning apparatus 544 can be activated to perform a data capturingphase 550 at certain times of the day, or during certain time intervals.

In embodiments where the scanning apparatus 544 is engaged when the door526 is moved to the closed position 530, it is contemplated that thescanning apparatus 544 may not be activated until a certain amount oftime has passed after the door 526 has closed. When a user is accessingthe refrigerating appliance 510, the user often opens and closes thedoor 526 of the appliance 510 repeatedly to access multiple temporarilystored items 542 within the refrigerating compartment 518. The scanningapparatus 544 can be configured to wait a predetermined amount of timealong the order of seconds, minutes, 15 minutes, 30 minutes, or more,before activating the next data capturing phase 550. This delay in theactivation of the scanning apparatus 544 can help to insure that theuser is likely done manipulating the various temporarily stored items542 with respect to a particular meal, so that an accurate, orsubstantially accurate, current inventory manifest 560 can be derived.It is contemplated that a delay in the scanning apparatus 544 can beuseful where certain items may take various amounts of time to settle orfind their level. Such viscous items such as mayonnaise, mustard, syrupand other similar viscous items may stick to the sides of the respectivecontainer 578, such that an accurate visual estimation of the remainingcontents 580 of the container 578 can be difficult, until such time asthe contents of the container 578 has settled and found its level.

It is contemplated that the object recognition system 672 that includesthe scanning apparatus 544 can perform a data capturing phase 550 whenthe doors 526 are in the closed position 530. In this manner, lightingwithin the interior 572 of the appliance 510 can be activated so thatthe one or more data capturing phases 550 can be conducted. The lightingcan be the general lighting for the interior 572 of the appliance 510.Dedicated lighting can also be included that is configured to work inconjunction with the scanning apparatus 544 to best illuminate theinterior 572 and the temporarily stored items 542.

It is also contemplated that the scanning apparatus 544 can be activatedwhen the door 526 is moved to the closed position 530, without asubstantial delay in any respect. In this manner, substantiallyreal-time information can be delivered to the user in the form of anup-to-date current inventory manifest 560 of the appliance 510.Accordingly, information concerning the temporarily stored items 542within the appliance 510 can be delivered as the various temporarilystored items 542 are used.

According to the various embodiments, the scanning apparatus 544 canalso include a manual or at least semi-automated input function wherecertain items that may not be recognized by visual scanning or weightscanning can be input. Such items can include food items (i.e.,leftovers stored within a substantially opaque or translucent storagecontainer 610) or other similar items that may not be visuallydiscernable. In such an instance, the scanning apparatus 544 can notifythe user of an unidentifiable item and may prompt the user for manualinput concerning the remaining contents 580 thereof. It is alsocontemplated that various storage containers 610 can include identifiertags 612 that work in conjunction with the scanning apparatus 544 wheresuch tags 612 can be relate to various information concerning thecontents of the storage container 610.

By way of example, and not limitation, after a particular food item isdone being prepared and is ready for storage within the refrigeratedappliance 510, the user can input a certain tag 612 or attach a certaintag 612 onto the storage container 610. Various identifying tags 612 canbe useful in a situation where a number of storage containers 610 havinga similar appearance are contained within the refrigerated compartment520. The identifier tags 612 can be attached to the opaque storagecontainers 610 such that each storage container 610 can be tracked as itis stored within the refrigerating appliance 510. Accordingly, onestorage container 610 having a vegetable stored therein and anothersimilarly shaped opaque storage container 610 containing a meat can betracked as each storage container 610 is moved or shifted within therefrigerating appliance 510 during use of the appliance 510. The manualinput can be performed by various manual input mechanisms that caninclude, but are not limited to, key pads, voice recognition devices,hand scanners, selection menus, query-based inputs, combinationsthereof, and other similar input devices.

According to various embodiments, as exemplified in FIGS. 1 and 2, it iscontemplated that the scanning apparatus 544 of the appliance 510 canalso include an exterior scanner 620 that can be disposed on or within aportion of the appliance 510 or a hand held scanner that can be used tomanually scan temporarily stored items 542 that are to be stored withinor outside of the refrigerating compartment 518. This hand scanner 620can be a wireless scanner, corded scanner, or can be integrated within aportable computing device 630 (shown in FIG. 16), such as a cell phone,tablet, wearable computing device, or other similar portable computingdevice 630 that the user can implement to scan temporarily stored items542, or scan other items related to a household inventory of consumableand perishable items, as will be described more fully below. Theportable computing device 630 can include a non-transitory computerreadable medium that stores one or more programs and the graphical userinterface on a computing device with a touch screen display, a memory,and one or more processors to execute one or more programs stored in thememory, wherein the touch screen device is operatively connected to oneor more database servers via a wired or wireless network of the presentinvention.

According to the various embodiments, as exemplified in FIGS. 1, 2, 5-7and 17-19, the scanning apparatus 544 can include one or more cameras570 that are strategically located on the doors 526 of the refrigeratedcompartment 520 or on the cabinet 512 of the refrigerated compartment520. The cameras 570 can be oriented to face the inside of the appliance510 when the door 526 is in a closed position 530. In this manner, whenthe doors 526 are in the closed position 530, the cameras 570 can viewthe contents of the shelves 640 disposed within the interior 572 of therefrigerated compartment 520. It is contemplated that cameras 570 mayonly be disposed on the doors 526 such that when the doors 526 areclosed, the cameras 570 are specifically and synergistically positioned,to be able to capture views of most, if not all, of the temporarilystored items 542 contained within the refrigerating appliance 510. It isalso contemplated that the configuration of the shelves 640 can assistin capturing images of the temporarily stored items 542 to get anaccurate picture of the current inventory manifest 560. Such shelfconfigurations can include a U-shaped shelf 640 where items disposed inthe various shelves 640 are generally not more than a few inches awayfrom an outer edge of the shelf 640 in a pantry-like configurationwithin the refrigerating compartment 518. In this manner, each of thetemporarily stored items 542 contained within the refrigeratingcompartment 518 are substantially visible by the various cameras 570 ofthe scanning apparatus 544 for viewing contents of the refrigeratingcompartment 518, as well as the positioning of the plurality of storagesurfaces 540 and the positioning, contents, and status of thetemporarily stored items 542.

Referring to FIGS. 5, 7, 17, 18 and 19-23, to further assist capturingdata during the first and second data capturing phases 552, 554, theshelves 640 can also include clear storage surfaces 540 such that acamera disposed on the door 526 can view through a transparent shelf 640at a particular temporarily stored item 542 disposed below thetransparent shelf 640. According to various aspects of the device, it iscontemplated that numerous shelf configurations can be included withinthe refrigerated compartment 520. Such shelf configurations can include,but are not limited to, U-shaped shelves, narrow-profile shelves,staggered-height shelves, rectangular shelves, height-adjustableshelves, shelves having slidable partitions, transparent shelves,translucent shelves, at least partially translucent shelves, opaqueshelves, shelves having integrated lighting, combinations thereof, andother similar configurations for allowing a camera to capture dataand/or images as to all or substantially all of the temporarily storeditems 542 contained within the refrigerating compartment 518. It isfurther contemplated that the cameras 570 and/or the shelves 640 can beoperable in a variety of directions (i.e., vertically, horizontally,laterally, diagonally, arcuately, irregularly, etc.) in order to providea wide range of views for performing the first and second data capturingphases 552, 554 to accurately gather information as to the temporarilystored items 542 to generate the current inventory manifest 560.

Referring again to FIGS. 5 and 7, it is also contemplated that variousdrawers 642 contained within the refrigerating compartment 518 caninclude transparent or at least partially transparent materials suchthat a camera 570 of the scanning apparatus 544 can view items containedwithin various drawers 642 of the appliance 510. Such drawers 642 caninclude, but are not limited to, crisper drawers, pantry drawers,drawers for humidity-controlled compartments, heating compartments,warmers, and other similar drawers 642 of the appliance 510. The variousdata capturing phases 550 for drawers 642 of an appliance 510 can beconducted as the drawer 642 is moved between a closed position 530 andan open position 528 where the drawer 642 is extended from therefrigerating compartment 518.

In order to capture images of all of the various compartments of theappliance 510, it is contemplated that various cameras 570 can bedisposed throughout the appliance 510 for capturing and analyzing dataand performing the first and second data capturing phases 552, 554within each of the compartments. Such compartments can include, but arenot limited to, refrigerated compartments 520, pantry compartments 524,freezer compartments 522, heaters, warmers, and other similarcompartments.

It is contemplated that each compartment can contain its own scanningapparatus 544 that can be modified for the most efficient use withineach of the particular compartments. By way of example, and notlimitation, the scanning device disposed within a freezer compartment522 may be different than a scanning device disposed within a pantrycompartment 524, warmer, or refrigerated compartment 520 based upondifferent environment conditions within each compartment. It is alsocontemplated that a larger compartment, such as a refrigeratedcompartment 520, can include multiple cameras 570, where multiplecameras 570 may be better suited for capturing the locations of thetemporarily stored items 542 through the entire refrigerated compartment520. It is contemplated that the overall configuration of the scanningapparatus 544 can be specifically designed depending upon variousfactors relating to the particular appliance 510. Such factors caninclude, but are not limited to, the number of compartments containedwithin the appliance 510, the type of compartments, the size ofcompartments, the type of shelving within a compartment, theconfiguration of the shelving within a compartment, the shelvingmaterial, the number of doors 526 of the appliance 510, the number ofdrawers 642 of the appliance 510, combinations thereof and other similarfactors. It is further contemplated that the scanning apparatus 544 canbe modified after sale to accommodate the specific habits and patternsof the user. Such habits and patterns can include, but are not limitedto, a desired temperature of a particular compartment, the number oftimes the compartment is accessed per day or per week, vacationingschedules, and other similar personal idiosyncrasies.

Referring again to FIGS. 1-7, where the scanning apparatus 544 includescameras 570 disposed within a rotationally operable door 526, that ishingedly connected to the cabinet 512, it is contemplated that each door526 of a French door refrigerator can be configured to take a picture ofthe opposing French door 526 as the doors 526 are moved to a fully openposition 528. In this manner, the system can be configured to be capableof sensing the speed of each door 526 as each door 526 moves between theopen and closed positions 528, 530. Such a sensing operation can be donethrough motion sensors 650, a rotational sensor disposed proximate adoor hinge 652, or other similar movement sensing apparatus orcombinations thereof.

The speed sensing process can also be done using a sensor 650 integratedinto the camera or other portion of the scanning apparatus 544 todetermine the best possible time to conduct a data scanning phase as toan opposing French door 526 of the appliance 510. Typically, where eachof the French doors 526 is in a parallel configuration with respect toone another, and in an open position 528, the door interior 654 of eachFrench door 526 faces the interior 572 of the opposing French door 526.In this manner, cameras 570 disposed on or within a portion of each door526 can perform a data capturing phase 550 as to the opposing doorinterior 654 for gathering information related to the storage surfaces540 of each door 526 as well as the temporarily stored items 542disposed within each door 526. At the appropriate time after the doors526 are moved to the closed position 530, all of the data recordedduring the various data capturing phases 550 can be combined, analyzedand compared to determine a current inventory manifest 560 as to theentire appliance 510. It is also contemplated that more numerous scanscan be conducted as to the door interiors 654 than as to the interior572 of the refrigerated compartment 520. Such a configuration can beuseful where the user of the appliance 510 accesses the door interiors654 more than any other portion of the appliance 510. Such aconfiguration can be set during manufacture. This configuration can alsobe set by the user depending on the habits, patterns, and idiosyncrasiesof the user of the appliance 510 allowing for more personalization ofthe user(s) based upon their preferences. While French doors 526 arediscussed in this aspect of the device, it is contemplated that aside-by-side refrigerating appliance 510 can also include such anaspect, for capturing data related to the temporarily stored items 542in each of the refrigerator and freezer doors 526.

According to the various embodiments, as exemplified in FIGS. 6 and 7,it is contemplated that the cameras 570, when disposed on the doors 526of the appliance 510, can be used to capture a three-dimensional imageand/or a panoramic two-dimensional image of the contents of therefrigerated compartment 520. In this manner, as the doors 526 arerotated from the closed position 530 to the open position 528, or viceversa, a series of successive scans 660 that make up a data capturingphase 550 can be performed as the doors 526 move between the open andclosed positions 528, 530. In this manner, the perspective of eachcamera of the scanning apparatus 544 will be slightly moved during eachsuccessive scan 660 of the data capturing phase 550, such that, when thedata is stitched, combined, or otherwise aggregated, the resulting datarepresents a three-dimensional image and/or a panoramic two-dimensionalimage of the interior 572 of the refrigerating compartment 518. Thepoints at which each data capturing phase 550 is performed during themovement of the doors 526 can be dictated by a predetermined rotationalposition of each door 526 relative to the open and/or closed position528, 530 as the door 526 is moved. In such an embodiment, rotationalsensors within the door 526 hinges can monitor the rotational positionsof the doors 526 for conducting the series of successive scans 660.Accordingly, whether the door 526 is open quickly or slowly, the datacapturing phases 550 can be taken at substantially the same rotationalangles in order to create substantially similar three-dimensional and/ortwo-dimensional data images for providing an efficient comparison andfor providing an accurate data representation of the current inventorymanifest 560.

It is also contemplated that the cameras 570 can be configured tocapture video as the doors 526 are moved between the open and closedpositions 528, 530. It is contemplated that video recorded, according tovarious embodiments, can then be converted from video to still images tocreate either two-dimensional or three-dimensional images for generatingthe current inventory manifest 560 of the appliance 510.

According to these various configurations of the scanning apparatus 544,it is contemplated that the number of cameras 570 or other datacapturing points can be substantially limited. Accordingly, it iscontemplated that only a portion of the compartments of the appliance510 may be fitted with a scanning apparatus 544. It is contemplated thatthe scanning apparatus 544, according to various embodiments, may befitted only in portions of the appliance 510 that are accessed mostfrequently.

Referring again to FIG. 8, in order to arrive at the current inventorymanifest 560, the appliance 510 can include a processor 670 thatcooperates with the object recognition system 672 and receives, stores,modifies, processes and compares all of the data recorded during thefirst and second data capturing phases 552, 554. It is contemplated thateach of the first and second data capturing phases 552, 554 can includenumerous sub-phases, such as where a three-dimensional image isgenerated for calculating the current inventory manifest 560.Accordingly, each of the first and second data capturing phases 552, 554represents a particular data gathering process whereby data, as to thetemporarily stored items 542 and other contents of the appliance 510,doors 526, and various compartments and subcompartments of the appliance510, can be obtained, copied or recorded for producing a currentsnapshot of the contents of the appliance 510. These data capturingphases 550 can be compared to a previous “snapshot” of the contents ofthe appliance 510 that was recorded during the first data capturingphase 552 to form the current inventory manifest 560.

In addition to visual scanning through color or black and white cameras570, alternate imaging techniques can be implemented during the firstand second data capturing phases 552, 554. Such imaging techniques caninclude bar code scanning, thermal scanning, infrared scanning, laserscanning, sonic or acoustical scanning, echo location, and other similarscanning/imaging techniques.

Referring again to FIGS. 1-7 and 17-23, where a U-shaped shelf 640 isinstalled within the refrigerated compartment 520, the temporarilystored items 542 disposed on each U-shaped shelf 640 can be placed suchthat they are continually near an outside perimeter of the U-shapedshelf 640. An operable large clear tuck shelf 640 can also be disposedin the center, where additional space is needed. By organizing itemsaround the perimeter of the U-shaped shelf 640, maximum viewing anglesare created to assist the scanning apparatus 544 and recording dataconcerning the stored items disposed within the refrigeratingcompartment 518 creating synergistic results for the system of thepresent disclosure. Additionally, the U-shaped configuration providesfor placement of the temporarily stored items 542 that produces aminimal risk of visual blocking of certain items from the variousportions of the scanning apparatus 544. By providing for the storage ofthe temporarily stored items 542 on the peripheral of each of theU-shaped shelves 640, the shelf 640 can help guide consumers to placeitems at an appropriate location and at a convenient and efficientdistance from the cameras 570 for optimal viewing by the scanningapparatus 544. This effect creates maximum views, fuller views andclearer views of more of the temporarily stored items 542 within therefrigerating appliance 510. This configuration also creates betterviews as to each of the shelves 640, such that visual blockage in avertical direction can also be minimized. According to variousembodiments, it is also contemplated that the scanning apparatus 544 canbe operable within the appliance 510 such that the various cameras 570,scanners 620, and other components of the scanning apparatus 544 can beoperable within the appliance 510 for gathering a maximum amount ofinformation concerning the various storage surfaces 540 and temporarilystored items 542 disposed thereon or therein.

Referring now to FIG. 20, a front view of a FDBM refrigerator isexemplified with the refrigerator doors 526 open. In the refrigeratorcabinet 512 shown, there are three U-shaped shelves, respectivelylabeled 101, 102 and 103. Also shown on each of the shelves is a raisedarea 104 around the perimeter of each of the shelves 101-103. Whenutilized, the raised area 104 prevents spills from coming down fromedges of the sides, back or inside of the refrigerator cabinet 512. TheU-shaped shelves 101-103 could be mounted on ribs or guides which extendoutwardly from the visible surface of the refrigerator cabinet 512.Alternatively, there could be recesses in the cabinet 512 withprojections on the shelves 101-103 sliding into the recesses, or theshelves 101-103 could be mounted on slidable brackets. However, theexemplary embodiments are not limited thereto and other known methods ofattaching the U-shaped shelves 101-103 to the inside of the refrigeratormay be used, as would be understood by one of ordinary skill in the art.It may be possible to mount the shelves 101-103 of the presentdisclosure to both sides and the rear wall and not utilize a centermounting bracket/system as is often used currently to support shelvesthat only extend across a maximum of one-half the width of therefrigerating compartment 518. The shelves of the present disclosuretypically extend across the entire width or substantially the entirewidth of the refrigerating compartment 518. The shelves 101-103 of theexemplary embodiments provide a panoramic view. Any of the shelves101-103 or all of the shelves 101-103 can have clear tuck shelvestherein or removable shelves that are simply slidable out of the U-shapeor lifted out of the U-shape by lifting the removable shelf upwards,typically straight up. A bottom U-shaped shelf is typically the same astop U-shaped shelf 111, 112 and has grooves therein 126, 129 forreceiving a tuck shelf that could be put into place and slid in underthe back of the U shape to provide more space for taller items belowU-shaped shelf 113, 114 in the refrigerator cabinet 512.

Referring now to FIGS. 21-23, a front view of a FDBM refrigerator isexemplified showing the French doors 526 in an open position 528. Inthis exemplary embodiment, three U-shaped shelves are shown. The openingin the U-shaped shelf 108 can be shifted over to one side. This recessand the other recesses could either have a removable shelf 106, a tuckshelf 107, or no shelves at all. In addition, the various shelves forselectively occupying the various recess configurations can be made ofglass, plastic, wood, a wood-looking material, combinations thereof, orother similar material.

As exemplified in FIG. 23, two tuck shelves are included within U-shapedshelves 111, 112 and 115, 116. A top U-shaped shelf 111, 112 may have afirst tuck shelf 105 that can slide in a horizontal manner under theback of the U shaped shelf 111, 112. The second U-shaped shelf 115, 116has a tuck shelf 107. In addition, middle U-shaped shelf 113, 114 mayhave a removable shelf 106 therein. The removable shelf may be slid orlifted out of the U-shaped shelf. In addition, the removable shelf 106,as shown, fills the U-shape of the U-shaped shelf 115, 116 but can beconfigured to fill less than all of the U-shape of this U-shaped shelfwhen in the extracted position. In addition, tuck shelves 105 and 107can be replaced with removable shelves. Alternatively, there may be noshelves within the center portions of the U-shaped shelves 111-116 ofthis exemplary embodiment.

According to the various embodiments, it is contemplated that aparticular data scanning phase can incorporate multiple components. Byway of example, and not limitation, an initial portion of the datacapturing phase 550 may be conducted to determine substantial changes inthe configuration of the storage surfaces 540 and the varioustemporarily stored items 542 within the refrigerated compartment 520. Itis contemplated that where no substantial changes have been made, suchas when an individual opens the refrigerator door 526 and closes thedoor 526 a very brief time thereafter, a lower resolution scan may beconducted. Additionally, the refrigerating appliance 510 may includemotion sensors 650 (shown in FIG. 1) that can detect movement within therefrigerated compartment 520 that can be activated when the doors 526are moved to the open position 528, in order to determine whether anymovement has been made to the storage surfaces 540 and/or thetemporarily stored items 542 of the appliance 510. In situations where auser simply opens the door 526 and views the contents without actuallyengaging any of the one or more temporarily stored items 542, the motionsensors 650, when detecting no movement, can instruct the scanningapparatus 544 to refrain from conducting a data capturing phase 550, asno substantial changes may have been made since the previous datacapturing phase 550. Accordingly, the motion sensors 650 can beconfigured to detect when an individual places their hand within therefrigerating compartment 518, or within a portion of the door 526 ofthe appliance 510, to access one or more of the temporarily stored items542. Where no such movement is detected, the scanning apparatus 544 canbe instructed, such as by a processor 670 of the appliance 510, themotion sensors 650, or other control mechanism, to refrain fromundertaking a data capturing phase 550. Alternatively, the scanningapparatus 544 can be instructed to undertake a low level or lowresolution data capturing phase 550. Such motion sensors 650 caninclude, but are not limited by, door switches, proximity sensors,infrared sensors, lasers, acoustical motion sensors, combinationsthereof and other similar motion sensing devices.

It is also contemplated that the scanning apparatus 544 can beinstructed to undertake a data capturing phase 550, each time the door526 is being moved to the closed position 530, whether the interior 572of the refrigerated compartment 520 has been accessed or not.

According to the various embodiments, as exemplified in FIGS. 8-16, thecurrent inventory manifest 560 is derived through use of the scanningapparatus 544 with a listing of their current inventory contained withinthe appliance 510. Such an inventory can be provided to the user througha read-out on the appliance 510 itself, or through various wirelesscommunications transferred from the appliance 510 to an exteriorcomputing device such as a personal computer or a mobile computingdevice with a touch screen display, memory and one or more processors.In this manner, data regarding the current inventory manifest 560 of theappliance 510 can be delivered via email, text, status update, ping, orother digital notification to a user's portable computing device 630.Such a portable computing device 630 can include, but is not limited to,a cell phone, smart phone, tablet, laptop, desktop computer, wearablecomputing device, or other similar computing device. It is alsocontemplated that data from the data capturing phases 550 and theinformation concerning the current inventory manifest 560 of theappliance 510 can be delivered to a “cloud” storage location for easyaccess by the object recognition system 672 and/or the user through anyone or more of the portable or stationary computing devices describedabove. Accordingly, because the current inventory manifest 560 of theappliance 510 is continually accessible by the user, the user may notneed to sort through the various temporarily stored items 542 containedwithin the appliance 510 to determine what items are within theappliance 510. The current inventory manifest 560 can be displayed as anitemized, categorized, or otherwise organized list. Such organization ofthe list can be dictated by the user based upon their needs at aparticular time.

As discussed above, the portable computing device 630 can include anon-transitory computer readable medium that stores one or more programsand the graphical user interface on a computing device with a touchscreen display, a memory, and one or more processors to execute one ormore programs stored in the memory, wherein the touch screen device isoperatively connected to one or more database servers via a wired orwireless network of the present invention.

By way of example, and not limitation, a user standing near a meatdepartment of a grocery retailer can conveniently access the currentinventory manifest 560. The user can then categorize the currentinventory manifest 560 to display those meat products, meat-relateditems and items that can be prepared with meat products for variousrecipes. It is contemplated that a myriad of category configurations canbe used to display all or a portion of the current inventory manifest560 to allow the user to most efficiently and conveniently access theinformation.

In addition to the current information regarding the temporarily storeditems 542, the current inventory manifest 560 can also include dataconcerning the history of a particular temporarily stored item 542.Accordingly, an inventory management system 680 can be provided inconjunction with the current inventory manifest 560 to provideinformation about any one or more of the temporarily stored items 542within the appliance 510. Such information can include, but is notlimited to, the date that the item was originally purchased and/orplaced within the appliance 510, information regarding how many timesthe particular temporarily stored item 542 has been taken out and putback into the appliance 510, the amount of time the temporarily storeditem was not spaced within the appliance, and other various informationconcerning the history of a particular temporarily stored item 542. Suchinformation can allow the user to understand how quickly a particularitem is being used or consumed. Accordingly, such information can informthe user as to what size of a particular consumable or perishable itemthey should be purchasing to meet their needs. Additionally, suchinformation can be compared with a recorded expiration date of theparticular temporarily stored item 542 to determine whether items areclose to being spoiled or have spoiled. The information about the amountof time an item is not spaced within the refrigerator may also be usedto warn a user about spoilage that may have occurred simply from beingaway from a refrigerated or freezing chamber for too long of a timeperiod. A system for detecting spoilage and alerting the user will bedescribed more fully below.

According to the various embodiments, by recording the number of timesthat an item has been taken out of the refrigerator and replaced, suchinformation can be helpful in determining the contents remaining withinthe container 578 of the temporarily stored item 542. This can beparticularly helpful where the temporarily stored item 542 includes anopaque or only partially translucent container 578. By way of example,and not limitation, the user cannot readily see the remaining contents580 of a container 578 of sour cream that has been put back into therefrigerator. However, information that the sour cream has been takenout and put back into the refrigerator a total of 15 times can be usefulin assessing how much sour cream remains in the container 578. Asdiscussed above, the scanning apparatus 544 can also include a scale 590in one or more of the storage surfaces 540 that can also assist inassessing the remaining contents 580 of a particular temporarily storeditem 542. However, where no scale 590 is present, the number of timesthat a particular temporarily stored item 542 has been taken out andplaced back in the appliance 510 can allow the user to make judgement,and/or allow the object recognition system 672 to propose arecommendation, as to the remaining contents 580 within the container578 of a temporarily stored item 542.

Additionally, according to the various embodiments, when the scanningapparatus 544 records or tags a date when the item was first placed inthe refrigerator, the user can use this information to make anassessment as to whether an item is spoiled or not. Typically, certaintemporarily stored items 542 spoil faster than others, i.e., milk mayspoil faster than mustard. Additionally, in the case of a storagecontainer 610 that may contain leftovers from a previously preparedmeal, information regarding the date that the storage container 610 wasplaced in the refrigerator can inform the user about how old theleftovers contained therein might be. For example, the user can beinformed that a container 578 of leftovers was placed in therefrigerator seven days previously. Based upon this information, theuser can either be prompted with a recommendation to consume or disposeof the leftovers, or the user can simply make an assessment on their ownas to what to do with the leftovers contained within the particularstorage container 610.

It is contemplated that, where numerous storage containers 610 that maylook identical are placed within the refrigerator, individual anddistinct identifier tags 612 may be contained within or on a portion ofa storage container 610. Alternatively, tags 612 can be attached afterpurchase of the particular storage container 610. It is contemplatedthat a system of storage containers 610 having tags 612 can includedigital tags 612 for identifying one storage container 610 relative toanother. Such tags 612 can be indicia that can be scanned by thescanning apparatus 544 of the appliance 510. Alternatively, theappliance 510 can include a radio-frequency identification (RFID) readersystem that can work in conjunction with the system of tags 612, such asRFID tags, of the storage containers 610 to identify one storagecontainer 610 relative to another and also identify the contents of onestorage container 610 relative to another. Where an RFID or otherwireless or digital tag 612 is implemented, information concerning thecontents of a particular storage container 610 having an RFID tag 612can be input or modified either at a control for the appliance 510, orthrough a portable computing device 630 of the user. The tag 612 foreach of the storage containers 610 for holding various food or otherperishable items can include various digital, physical, visual or otherindicia that can be scanned through the scanning apparatus 544 of theappliance 510. In such an embodiment, such tags 612 can include, but arenot limited to, bar codes, an applied indicia (i.e., number, letter,symbol, color, combinations thereof, etc.), a physical shape of thecontainer 578 or a physical shape of a portion of the container 578(i.e., a shape of a handle, a raised relief, an indented portion,combinations thereof, or other similar physical feature of the storagecontainer 610).

According to the various embodiments, the scanning apparatus 544 caninclude various input methods for identifying what temporarily storeditems 542 are contained within the appliance 510. These methods can beaccomplished by various components of the scanning apparatus 544, wheresuch components can include, but are not limited to, cameras 570,scanners 620, visual sensors, acoustic sensors, light sensors, scales590, hand scanners, infrared scanners, infrared cameras 570, laserscanners, RFID readers, 3-D imaging, tactile sensors, GPS, and othersimilar sensors that can be used to determine various status, location,life span, and other information concerning the various temporarilystored items 542. Through these various components of the scanningapparatus 544, various methods for importing the identity of one or moretemporarily stored items 542 can include, but is not limited to,scanning an item to be placed within the appliance 510, photographing orotherwise imaging each temporarily stored item 542 to be placed withinthe appliance 510, scanning a container 578 for a temporarily storeditem 542 when the container 578 is empty or about to be emptied, acustomer-initiated scan or photograph of a temporarily stored item 542as each item is removed from or placed in the appliance 510, scanning areceipt 690 from a store (described more fully below), combinationsthereof, and other similar data gathering and optional analyzingmethods.

According to the various embodiments, as exemplified in FIGS. 1-17, itis contemplated that the scanning apparatus 544 can be integrated withone or more grocery retail stores 710 or other retailers such that thecurrent inventory manifest 560 can be compared with the inventory,prices, or used in conjunction with “frequent shopper” or rewardsprograms of one or more retail stores 710. In this manner, the currentinventory manifest 560 can be compared with such retailer data toformulate a shopping list 712 for the user based upon retail store 710location, relative prices, dates of sales, and other information suchthat shopping trips can be organized for the most efficient use of theuser's time. In addition, this information can be combined with theuser's own schedule, such that if the user is running an errand in aparticular geographic location, the user can be reminded to stop by anearby retail store 710 to fill a portion of the shopping list 712generated by a combination of the current inventory manifest 560 of theappliance 510 and information regarding that particular retail store710. Additionally, as a particular user is walking through the retailstore 710, information concerning items contained within a particularaisle can be presented to the user such that a user can be reminded tolook for a certain item in a particular aisle. Such a reminder from theobject recognition system 672 can avoid aimless wandering through vastaisles of retail space and can generate a more efficient shopping trip.

In order to monitor a user's position within a retail store 710 inrelation to various products within a particular aisle, the objectrecognition system 672 can work in conjunction with various informationnetworks integrated into the retail store 710. One such network is anindoor positioning system (IPS) that provides information relating tothe locations of people and objects within the retail store using radiowaves, magnetic fields, acoustic signals, or other sensory informationcollected by portable computing devices 630. The IPS can also utilizewireless technologies, such as bluetooth, WiFi, etc., to transfer thedata to the user. One such IPS is iBeacon™, although others arecontemplated. Other indoor wayfinding solutions include, but are notlimited to, micromapping, indoor mapping, indoor proximity systems, andothers. These IPS systems and similar systems can communicate with theuser's portable computing device 630, inertial measurement units,wearable computing devices and other devices incorporating a positioningcapability, and work in conjunction with the current inventory manifest560, shopping list 712, comprehensive menu profiles 740 (describedbelow) and other components of the object recognition system 672.

According to the various embodiments, the scanning apparatus 544 of theappliance 510 can include an imaging system having resolution providingenough detail to capture labels of particular temporarily stored items542, bar code information, and other detailed information concerningparticular temporarily stored items 542 purchased from various retailstores 710. This label information can be stored to assist in theformation and maintenance of the current inventory manifest 560.

In conducting each of the data capturing phases 550, it is contemplatedthat different colors of light and different types of light can beilluminated during various subroutines of the data capturing phase 550.Changes in the color of light such as between blue, green, red and othercolors, can reveal different characteristics concerning varioustemporarily stored items 542. Such use of colors can also allow theimaging system to record data through translucent containers 578 havingcolored materials. Additionally, different types of light (i.e.,fluorescent, incandescent, LED, halogen, combinations thereof, etc.) canhave an effect on how the temporarily stored item 542 is viewed by thescanning apparatus 544. Certain types of artificial light may travelthrough various containers 578 in a more efficient and effective mannerthan other types of artificial lighting. The use of alternate lightingcolors and lighting types can also help to distinguish betweentemporarily stored items 542 that may have a similar shape, orcontainers 578 for temporarily stored items 542 that may have a similarshape. The use of types of colors of light can also assist indetermining the state, such as in terms of spoilage, remaining contents580, and other information, of certain temporarily stored items 542. Inthis manner, the output related to the temporarily stored item 542 canbe a shopping list 712 delivered by the object recognition system 672 tothe user of a specified set of temporarily stored items 542 that need tobe replenished.

Referring now to various aspects of the device as illustrated in FIGS.1-11, the scanning apparatus 544 of the appliance 510 can be used togenerate the current inventory manifest 560. The current inventorymanifest 560 can be analyzed, and, based on the status of any one ormore temporarily stored items 542, a shopping list 712 for the user andthe entire household can be generated based on the current inventorymanifest 560. It is contemplated that the current inventory manifest 560can include information concerning the amount of a particulartemporarily stored item 542 that is remaining, the life span of thetemporarily stored item 542, the date that the temporarily stored item542 was disposed within the appliance 510, and other informationconcerning the life span of a particular temporarily stored item 542.According to the various embodiments, when a particular temporarilystored item 542 is about to be fully used or consumed, the item 542 canbe automatically added to a shopping list 712 that can be delivered tothe user. The shopping list 712 can be stored either within theappliance 510, within a user's portable computing device 630, or withina “cloud” data storage computer system accessible via a wired orwireless network.

Referring now to FIG. 13, according to the various embodiments, where aparticular temporarily stored item 542 is about to reach its designatedexpiration date, the temporarily stored item 542 can be automaticallyadded to the shopping list 712. Additionally, the user can be promptedto use the remaining amount of the temporarily stored item 542 before acertain date or to dispose of the particular item if the expiration datehas been reached or exceeded. In connection with such a prompt orrecommendation, various recipes can be suggested to the user thatincorporate the particular temporarily stored item 542 that is about toexceed its expiration date.

Additionally, in the case of fruits and vegetables, as exemplified inFIG. 13, visual scanning can be utilized to assess any bruising, oldspots, mold or other surface conditions of the food item that may beindicative of that particularly stored item exceeding its useful life.When such an indication is apparent, the user can be provided with aprompt or recommendation to use or dispose of the particular temporarilystored item 542, and the item can also be added to the shopping list712.

It is also contemplated that where a particular temporarily stored item542 is about to reach its expiration date without being substantiallyconsumed or otherwise used, the user can be prompted with a question asto whether or not to add the particular temporarily stored item 542 tothe shopping list 712, since the item was previously purchased and notsubstantially consumed or otherwise used. By way of example, and notlimitation, where 10 bananas are purchased, but only a minimal number ornone of the bananas are used or otherwise consumed, the objectrecognition system 672 incorporating the scanning apparatus 544 can askthe user whether bananas should be added to the shopping list 712 and/orthat fewer bananas or no bananas be purchased during the next shoppingtrip.

Referring again to FIG. 10, after the shopping list 712 is developedusing the current inventory manifest 560, the object recognition system672 can work in conjunction with various retailer stores, retailer websites, and other similar sources of information as to retail prices ofvarious retail stores 710, sales provided by retail stores 710, andother pricing information of various retail stores 710 within the user'sarea. Accordingly, as discussed above, a schedule of shopping trips canbe suggested to the user based on particular prices, sales, the user'sscheduled events for a particular time period, and other information tomake each shopping trip as efficient as possible. Additionally, wherepaper coupons are necessary during a certain shopping trip, the objectrecognition system 672 can cooperate with a local or network printer viaa wired or wireless network connection to automatically print outvarious coupons that correspond to the current inventory manifest 560and/or the shopping list 712 presented to or accessible by the user.Additionally, the shopping list 712 can be communicated to a grocerydelivery service where the delivery service receives the shopping list712 prepared by the object recognition system 672 and delivers thenecessary groceries to the user's household. Before such a shopping list712 is delivered, the user may have the opportunity to approve items onthe shopping list 712 or require that a certain expected cost for theitems be reached. The user can also request a maximum amount to bespent, particular quality of products (i.e., organic, locally produced,brand, style, retailer, appearance) and other requested details that theuser may take into consideration if they were to undertake the task ofshopping themselves. Various preferences of the user can be recordedwithin the object recognition system 672 and saved for later use.

It is also contemplated that the object recognition system 672incorporating the sensing apparatus can be used in conjunction withvarious calendars, weather information, event calendars, holidayschedules, daily schedules, and other dates, events and occurrences thatmay occur during a given time period. According to various embodiments,it is contemplated that the object recognition system 672 can operate inconjunction with the various weather reports to suggest the purchase ofitems that may be needed during a storm, during a particularly severewarning having an appreciable advance notice such as a hurricane,thunderstorm, heat wave, snow storm, period of extreme cold, or othersimilar weather occurrence. Various other items can be suggested foradditions to the shopping list 712 that may be needed during or afterthe particular weather occurrence. By way of example, and notlimitation, in a region that may receive a hurricane warning, the objectrecognition system 672 can suggest adding various items that may beneeded during or after the hurricane, such as, but not limited to,batteries, consumable bathroom products, generators, fresh water,portable grills, and other similar items that may be useful inweathering and recovering from a particular weather event.

The object recognition system 672 can also work in conjunction withvarious scheduled events such as holidays, sporting events, or scheduledgatherings. By way of example, and not limitation, where a birthday of amember of a household is scheduled, certain items can be suggested foraddition to the shopping list 712 that may be useful or appreciatedduring the birthday party. Such items can include, but are not limitedto, cake mix, frosting, candles, wrapping paper, and other similar itemsthat may be desired during a party.

As previously discussed, items added to the shopping list 712, whetherregularly purchased items, items for special events or weather or otheremergency situations, the object recognition system 672 can coordinatethe shopping list 712 with information gathered concerning variousretailers to create an agenda or schedule for various shopping trips tospecific retailers for purchasing the items on the shopping list 712 andnotify the user that the item(s) are in stock by the retailer at thattime.

It is also contemplated, according to the various embodiments, that theobject recognition system 672 can compare the shopping list 712 againstvarious local events such as farmer's markets, various festivals, artfairs, and other similar events. In the case of a farmer's market, theshopping list 712 can suggest purchasing certain items generallyavailable at farmer's markets, especially items that are considered “inseason” for the geographic region at the time.

As discussed previously, the object recognition system 672 incorporatingthe scanning apparatus 544 can detect, record, and provide to the userinformation concerning the lifespan, expiration, spoilage, or otherlifespan related data with respect to one or more temporarily storeditems 542. A scanning assembly can record various information directlyoff from the temporarily stored item 542 related to an expiration date,or use-by date, that can be imprinted directly on the temporarily storeditem 542. The object recognition system 672 may also be configured torecognize when an item is about to or has spoiled, such as produce,dairy products, and other perishable items. With respect to produce, thescanning assembly can detect, through visual inspection, infraredinspection, inspection under various lighting colors or lighting types,and other imaging techniques, whether an item of produce has spoiled. Inthis manner, this scanning assembly can detect, over time, progressivebruising, discoloration, or other physical changes in produce that occurover time.

By way of example, and not limitation, as exemplified in FIG. 13, thescanning assembly may detect a small blemish 730, bruise or other markon the surface of an apple. Overtime, and during a subsequent datacapturing phase 550, the scanning assembly may detect that thediscolored area of the blemish 730 has enlarged. At such time, theobject recognition system 672 can alert the user that the apple isnearing its useful life and should be used or disposed of in the nearfuture. If the item remains within the appliance 510, the scanningassembly can continue to monitor the apple, and during a subsequentphase, can detect that the affected area of the blemish 730 has expandedover a significant portion of the apple. At this time, the objectrecognition system 672 can alert the user and recommend that the userdispose of the apple. As discussed previously, where food spoilageoccurs, the user can be prompted as to whether the item should bereplaced by being added to the shopping list 712. Because the item isspoiled, the object recognition system 672 can identify that thetemporarily stored item 542 was not used in a timely fashion, such thata smaller amount of the item 542 may be purchased, or that a replacementfor the temporarily stored item 542 may not need to be purchased at all.The object recognition system 672 can include similar functionality fordetermining when a certain produce item is ripe. For example, the objectrecognition system 672 can include a color recording camera that candetect that a banana has turned from green to yellow and is in primecondition for being consumed.

According to the various embodiments, with respect to containers 578 oftemporarily stored items 542, visual inspection can, as to sometemporarily stored items 542, visually detect a change in the container578 when a food approaches a spoilage state. By way of example, and notlimitation, when a container 578 of milk approaches a spoilage state,gasses within the container 578 can accumulate and cause the container578 to expand. This expansion can be detected by the scanning assemblyand communicated to the user in form of a prompt to verify the state ofthe milk, dispose of the milk, combinations thereof or other similarprompt to address the potential or actual spoilage of the container 578of milk. Additionally, it is contemplated that the scanning assembly caninclude air sensors that can monitor the state of the air containedwithin the refrigerating appliance 510. Such air sensors can include airquality sensors, pollution sensors, CO2 sensors, sensors for monitoringcontaminants indicative of spoilage, and other sensors. In this manner,when a temporarily stored item 542 becomes spoiled or nears a spoilagestate, various odors can be given off by the temporarily stored item542. The scanning apparatus 544 can include an air detection mechanismthat can detect such fumes and other matter in the air indicative of aspoiled food item.

According to the various embodiments, as a temporarily spoiled fooditems nears a spoilage state, the object recognition system 672 canalert the user to various recipes that can incorporate the temporarilystored item 542 that is nearing a spoilage state. Additionally, theobject recognition system 672 can also recommend recipes that may beprepared using the temporarily stored item 542 nearing the spoilagestate. If certain additional items are purchased, as discussed above,these items can be added to a shopping list 712, and the shopping list712 can be coordinated with various information from local retailers formanaging shopping excursions, as discussed herein.

Referring now to the aspects of the various embodiments as illustratedin FIGS. 1-11, the object recognition system 672 in incorporating thescanning apparatus 544 can be used for household recipe planning. Therecipe planning processes can be conducted for individual members of ahousehold or for the household in general, or both. According to thevarious embodiments, the refrigerating appliance 510 can include aset-up function that can be configured to receive information concerningthe various members of the household. Individual members of thehousehold can be added and specific preferences of each member can bespecifically identified. By way of example, and not limitation, ahousehold exemplified in FIG. 11 may contain Mom, Dad, Dick, Jane,Grandma and Grandpa. A selection for Grandma can bring up a menu-typefunction, wherein certain dietary preferences or restrictions can beidentified for each member of the family. For example, Grandma may beable to select one of several menu preferences including, but notlimited to, diabetic, vegan, vegetarian, specific dietary systems suchas Atkins®, South Beach® and other similar dietary preferences. Inaddition, various allergies or dietary restrictions can also be addedwith respect to Grandma's preferences, such that a comprehensive menuprofile 740 can be incorporated within the appliance 510 for generatingshopping lists 712 and performing the shopping functions related to theobject recognition system 672. As the current inventory manifest 560 isformulated through operation of the various data capturing phases 550 ofthe scanning apparatus 544, certain recipes can be suggested based uponthe current inventory manifest 560.

Additionally, recipes can be suggested based on current inventorymanifest 560 as well as the various comprehensive menu profiles 740 foreach household member. Where certain menu restrictions or preferencesare desired, ingredient substitutions can be recommended based upon thevarious menu restrictions. Accordingly, menu planning that accommodateseach comprehensive menu profile 740 and/or all of the comprehensive menuprofiles 740 for the various household members can be generated so thata single meal can be prepared for the entire household. Where certainmenu restrictions prevent everyone from eating such recipe, ingredientsubstitutions can be suggested that are more compatible with everyone'scomprehensive menu profile 740 within the household.

Also, it is contemplated that menu combination suggestions can beprovided where the combinations can coordinate based upon ingredients,available appliances, time constraints, and other similar factors. Forexample, suggestions can be made for the preparation of food items for aparticular meal that may be easily prepared in combination, such as onethat can be made within the oven and one that can be made on a stovetop,for simultaneous preparation of foods. In addition to the specific menutypes described above in formulating a comprehensive menu profile 740for each household member, certain calorie limitations or guidelines toassist the household member in maintaining the desired caloric intakeare provided. Additionally, various food profiles can also be generatedwhere a household member desires a certain percentage of protein,carbohydrate, fat, cholesterol, or other specific identified nutrientsin their diet on a daily basis.

According to the various embodiments, in order to assist each householdmember in maintaining a diet that incorporates the desired comprehensivemenu profile 740, the object recognition system 672 can be configured tosearch a database of recipes, or connected to various recipe-orientedwebsites for researching and recommending various menu items and recipesthat are compatible with the various comprehensive menu profiles 740 forthe various household members. Moreover, the recommended menu items andrecipes can also take into account the current inventory manifest 560,as well as the shopping list 712 generated using the current inventorymanifest 560, to provide for efficient use of items currently within thehousehold, or which will be obtained in the near future. These searchedrecipes can be modified or modifications can be suggested by the objectrecognition system 672 based upon the various comprehensive menuprofiles 740. Also, the recipes can be multiplied or dividedautomatically to be more compatible with the number of the people eatinga particular meal. By way of example, a recipe that serves four can beautomatically adjusted by the object recognition system 672 by 1.5 timesto be sufficient to serve a family of six.

According to the various embodiments, it is contemplated that the objectrecognition system 672 can incorporate a function for determining alevel of cooking skill for a particular household member. This level ofskill can be incorporated into the object recognition system 672 forproviding recommended recipes and menu items to be prepared by variousmembers of the household. An individual at a novice-cooking ability maybe provided a selection of recommended menu items having a low-level ofcooking skill required. Alternatively, an individual with a high-levelof cooking skill may be provided with recommendations for more complexrecipes that may require significantly more skill. The objectrecognition system 672 can take into account the current skill level ofan individual and assist the individual in improving the cookingability, such that recipes can be recommended that require increasinglevels of skill over time.

By way of example, and not limitation, an individual at a novice-typecooking skill level may be provided recommended recipes of comparativelyeasy to prepare menu items, such as sandwiches, salads, certain pastadishes, and other comparatively easy-to-prepare menu items. Over time,the object recognition system 672 can recommend increasingly morecomplex menu items that require additional skill to prepare. In order toassist the user in increasing their skill level in cooking, the objectrecognition system 672 can work in conjunction with various displayscreens for providing instructional videos for how to prepare variousmenu items. Step-by-step instructions may be provided and pre-recordeddemonstration videos can also be provided. Additionally, variousinternet sites can be researched and provided to the user for showingsample methods for preparing various food items. Over time, theindividual utilizing the various cooking techniques can learn newcooking techniques and methods so that their skill can increase overtime. As the cooking skill of the individual increases, more complexrecipes can be recommended to the user, where more complex recipes caninclude, but are not limited to, sauces, various roux recipes, marinadesand more gourmet-type recipes.

As an aspect of the object recognition system 672, as the user requiresadditional cooking skill, additional appliances may be needed forpreparing certain items, such as fresh pastas, ground items that requiremixers, blenders or other appliances. Suggestions for purchasing theseitems as an easier way of preparing more complex recipes can besuggested automatically (without user prompting) by the system(s) of thepresent disclosures to the user. It is contemplated that, in variousembodiments, the user may opt out of various messages/suggestions wherethe user may find such messages a nuisance.

The object recognition system 672 can also be programmed to be sensitiveto certain preferences of the user. By way of example, and notlimitation, where a user feels like having a “beef” recipe, a prompt ofthe object recognition system 672 can include a query of “what are youin the mood for eating?” Based upon the answer to this query, certainmenu items or recipes can be suggested to the user and based upon theuser's selection and the current inventory manifest 560. Various itemsfor preparing these recipes may also be added to the user's shoppinglist 712 for access by the user's portable computing device 630, or forprinting out for the user in preparation for the shopping excursion.Additionally, a section of the object recognition system 672 can includea file of a user's favorite recipes or most liked recipes. Where a userhas several favorite recipes, these recipes can be sequenced to limitover repetition of prepared meals. It is contemplated that certain “trysomething new” opportunities can be recommended through the course ofthe routine favorite meals so that the diversity of the user's menu canbe gradually increased. It is also contemplated that certain recipesthat are disliked can be deleted from the object recognition system 672.Future recommendations can also be adjusted based upon “dislike”preferences.

It is also contemplated that in connection with the object recognitionsystem 672, the menu and shopping functions and other functionalities ofthe object recognition system 672 can be used in conjunction withvarious services, internet subscriptions and other similar services thatcan be used in conjunction with the object recognition system 672 forsuggesting products, menu items, recipes, retail store offers, sales,and other services that can make shopping and preparing meals convenientand more efficient. Such services can include, but are not limited to,Weight Watchers™, Blue Apron™, Plated™, Hello Fresh™, and others.

It is further contemplated that the object recognition system 672 caninclude various environmental functionalities that may add to theexperience of preparing a meal. By way of example, and not limitation,the system can integrate with a kitchen speaker system (i.e., CoolVox™)and play music that may share a theme with a particular recipe. Whenpreparing a Mexican dish, a user may desire mariachi music to be playingin the background. Similarly, holiday-themed music can be played inconjunction with recipes prepared during corresponding holiday seasons.It is also contemplated that certain videos that may provide a specificambiance. For example, when preparing an Italian dish, a video from TheGodfather may be displayed in the background. Such environmental aspectsof the object recognition device can add to the experience and enjoymentof the cooking for creating more memorable events that can be shared byfriends and family.

According to the various embodiments, where a comprehensive profile canbe created for one or more users of the object recognition system 672,certain preferences can be changed, based upon a desired healthmanagement plan. Where a user desires to make a dietary change thataffects certain parameters within their body (i.e., weight, cholesterol,caloric intake, diabetic choices, recommended disease-fighting diets,combinations thereof and other health management choices), a change canbe made to an individual's comprehensive menu profile 740. As a result,recipe recommendations and shopping lists 712 can be modified to besensitive to these health management choices. Additionally, it iscontemplated that certain portion size suggestions can be recommended tomanage the proper caloric intake desired by the user.

The health management system can also be integrated with the waterdelivery system of the appliance 510 to deliver the water being consumedby various members of the household. This can be conducted through thewater dispensing system of the appliance 510 or through containers 578of water that can be stored within the refrigerated compartment 520. Inorder to keep track of a particular user's water intake, it iscontemplated that certain identifiers can be implemented to inform theobject recognition system 672 as to which user is accessing a portion ofthe appliance 510. It is contemplated that such an identification systemcan include signal communication with a smart phone, cell phone, awearable device, such as FitBit®, smart watch, other wearabletechnology, or other portable computing device 630. It is alsocontemplated that biometric indicators can be implemented in the form offingerprint scanners, voice recognition, or other similar uniqueidentifier that may distinguish one user from another. Where a user maynot be consuming enough fluid over the course of a day, politereminders, or customized reminders of varying intensity, can be providedto the user to drink more water throughout the course of a day.

It is contemplated that the object recognition system 672 for theappliance 510 can be incorporated with other appliances 510 as well. Forinstance, the appliance 510 can be placed in communication with a stoveor oven, where the particular cooking function and time remaining can becommunicated to the object recognition system 672. Where a particularrecipe requires a long cooking period, this time remaining in thecooking function can be communicated to the object recognition system672 or to the user directly, and a recommendation can be provided forexercising, walking or engaging in other activity while waiting for thecooking function to near completion. For example, when baking aparticular food item, the object recognition system 672 may suggest tothe user that they take a brief walk of approximately 10 minutes to passthe time while waiting for the cooking function to complete. Such signalcommunication between appliances 510 can also be implemented between theobject recognition system 672 and other appliances that can include, butare not limited to, blenders, mixers, ovens, ranges, washers, dryers,dishwashers and other similar appliances.

It is contemplated that changes can be made over time to the variousportions of the object recognition system 672 to reflect changes in thehousehold. For example, where a family has welcomed a new baby into thehousehold, the object recognition system 672 can be modified to reflectthe change by modifying the comprehensive menu profile 740 of abreast-feeding mother, a child, providing recommendations for recipesthat might be prepared more quickly, providing baby food reciperecommendations, and other similar changes that can be sensitive tochanges in the household.

It is also contemplated that the object recognition system 672 can bemodified to be sensitive to certain household events, such asparticularly busy work schedules, children's activity schedules, tragicevents in the household (i.e., death in the family, illness, injury,etc.) and celebrations (i.e., weddings, anniversaries, birthdays,vacations, etc.). Accordingly, the object recognition system 672 can bemodified to accommodate each of these personal events individually or incombination to modify the shopping list 712, recommended recipes, orother similar events to be sensitive to user's needs. By way of example,where a child has numerous athletic events such as baseball, football,gymnastics, ninja training, and other events that are occurring withinthe same time period, less time may be available for preparing meals.The object recognition system 672 can be sensitive to time constraintsand suggest recipes that require less time to prepare and may includefewer items such that meals can be prepared in a convenient andexpedient fashion so as not to take time away from family events. Duringsuch times, the object recognition system 672 can include recipes thatalso include proper nutrient values to be sensitive to the highercaloric needs of the people engaged in the various sporting orrecreational activities.

Referring now to FIGS. 1-15, it is contemplated that the objectrecognition system 672 cannot only maintain and assess a currentinventory manifest 560 for the appliance 510, but also of varioustemporarily stored items 542 and consumable products throughout theentire household. The object recognition system 672 can include ascanner 620 that may scan a receipt 690 or a series of receipts 690 thatreflect items purchased that may not be stored within the appliance 510and recognized by the object recognition system 672. Such items caninclude pantry items that may be stored within a cupboard or a roomtemperature pantry, bathroom consumable items (i.e., toilet tissue,soap, shampoo, etc.), various household consumables (i.e., air filters,light bulbs, laundry related consumables, dish washer relatedconsumables, and other household consumable products that are depletedovertime). By way of example, and not limitation, a scanner 620 can beassociated with the scanning apparatus 544, which can be a scanner 620disposed within or outside the appliance 510 or a portable computingdevice 630, can be used to scan a receipt 690 acquired after aparticular shopping trip to the retail store 710. Items included on thisreceipt 690 can be included onto the current inventory manifest 560 forthe household. Over time, certain temporarily stored items 542 for thehousehold may show a pattern of being purchased at regular intervalssuch that these items can be recommended to be added, or automaticallyadded, to the shopping list 712 in accordance with these regularintervals. It is also contemplated that each storage room for thehousehold where temporarily stored items 542 may be kept can have adedicated scanning apparatus 544 that is integrated as part of theobject recognition system 672.

For example, it is contemplated that in a household where a bottle oflaundry detergent is purchased approximately once a month, the objectrecognition system 672 can suggest adding laundry detergent to theshopping list 712 on a monthly basis to account for the consistent usesof a laundry detergent. Based on the pattern of usage of a particularconsumable or depletable item, the object recognition system 672 caninclude an approximate amount of a particular temporarily stored item542 contained outside of the refrigerating appliance 510 such that itemscan be added to the shopping list 712.

According to the various embodiments, various temporarily stored items542 can be suggested based upon the frequency of usage for a particulartemporarily stored item 542. Where a household includes a large numberof people and certain items such as laundry detergent are currentlypurchased in small to medium quantities, the object recognition system672 can suggest purchasing a “jumbo sized” laundry detergent from awholesale retailer, such as Costco or Sam's Club, to save money and alsolimit the number of times that laundry detergent is purchased throughthe course of a year. In this manner, the object recognition system 672can make suggestions as to shopping trips that correspond to sales ofcertain retailers and can also be organized upon sizes available for aparticular temporarily stored item 542. It is contemplated that objectrecognition system 672 can be configured to automatically calculate aprice per ounce or price per item so that a fair comparison betweenproducts can be analyzed to see if one product may be priced less on aper ounce or per item basis when compared to a similar differently sizeditem available at a different retailer.

According to the various embodiments, it is contemplated that the datastored through use of the object recognition system 672 can be availablefor use by the retail stores 710 in developing a strategy fordetermining what items to purchase in a particular locale. In such anembodiment, certain raw data related to purchases made by users in acertain geographic area (zip code, county, state, etc.) of the objectrecognition system 672 can be provided to retail stores 710. Theseretail stores 710 can use this data to discover various trends aroundspecific retail stores 710. Such trends could be helpful in assessingwhat products may be popular in one area but not another. Informationabout these trends can be useful to the retail stores 710 in making moreefficient purchasing decisions. Other data that may be useful caninclude items purchased, sizes of items purchased, numbers of aparticular item purchased, cooking skill level of residents in a certainregion, recipe likes and dislikes, preferred shopping locations, daysthat users go shopping, days that certain items are purchased overothers, and other information related to the purchasing decisions ofusers of the object recognition system 672. It is contemplated thatpersonal identifying information relating to the users of the objectrecognition system 672 may not be shared.

Referring again to FIGS. 1-15, it is contemplated that the objectrecognition system 672 can monitor the placement of various items withinone or more appliances 510. In this manner, the object recognitionsystem 672 can suggest more convenient or appropriate placement oftemporarily stored items 542 contained within the appliance 510. By wayof example, and not limitation, where the object recognition system 672senses that apples are placed on a top shelf 640 of a refrigeratedcompartment 520, the object recognition system 672 can suggest that theapples be moved where such placement may extend the life of thetemporarily stored item 542.

This can also be conducted between numerous refrigerating appliances 510contained within a certain household, where a household may havemultiple refrigerating appliances 510 spaced throughout the house, suchas a refrigerator in a kitchen, a mini fridge in the kitchenette, afreezer in the garage, and other potential refrigerating appliances 510,the refrigerating system may be placed in signal communication with eachof the refrigerating appliances 510. It is contemplated that each of therefrigerating appliances 510 can include a dedicated scanning apparatus544 for determining the locations of the storage surfaces 540 and thetemporarily stored items 542 disposed within each appliance 510,respectively. For purposes of placing certain temporarily stored items542 in the most convenient and efficient location within therefrigerating appliances 510, the object recognition system 672 cansuggest a particular location for various items purchased.

By way of example, and not limitation, if the primary refrigeratingappliance 510 in the kitchen is full and milk needs to be located in arefrigerating appliance 510, the object recognition system 672 cansuggest that the milk be placed in one of the other refrigeratingappliances 510 that has room to hold the temporarily stored item 542. Itis contemplated that the object recognition system 672 can be sensitiveto the storage space capacity within each of the refrigeratingappliances 510 and can modify the shopping list 712 accordingly toaccount for any space limitations that may be present according to thecurrent inventory manifest 560 of the entire household.

According to various embodiments, the capacity available within each ofthe refrigerating appliances 510 may be a conflict with a particularshopping list 712, such that items that need to be purchased may or maynot fit within the refrigerating appliances 510 based on the currentinventory manifest 560. The current object recognition system 672 cansuggest usage of certain temporarily stored items 542 or recipes thatincorporate certain temporarily stored items 542 that can use upremaining quantities of temporarily stored items 542 to open up spacesuch that shopping excursions can be conducted and the shopping lists712 can be filled to restock the house on certain items that may beneeded. In such a situation, the object recognition system 672 mayrecognize that certain items are very low in quantity and are nearingreplacement or needing replenishment. The object recognition system 672can utilize the current inventory manifest 560 to assess the amount ofspace that may be gained by using up these items in a particular recipeor through consumption (i.e., having a glass of milk, making an omelet,grilling hot dogs, etc.) where a single meal may use the remainingcontents 580 of these almost used temporarily stored items 542.

According to the various embodiments, the object recognition system 672can also assess various temperatures and humidity conditions of the airwithin a particular refrigerating appliance 510. Based upon the contentsof each compartment, the object recognition system 672 can suggestvarious modifications to temperature, humidity level, and otherparameters that may affect the longevity of a particular temporarilystored item 542 contained within the refrigerating appliance 510. Theenvironmental conditions of the interior 572 of the appliance 510 can becaptured through various sensors or monitors that can include, but arenot limited to, thermometers, humidistats, air flow sensors, and otherenvironmental sensors. It is also contemplated that the objectrecognition system 672 can recognize when certain containers 578 may benegatively affecting the longevity of certain temporarily stored items542. For example, the object recognition system 672 may recognize whencertain produce items being stored in a bag may negatively impact thelongevity of the produce item. The object recognition system 672 cansuggest removing the produce from the bag to allow the fresh produce tobreathe and allow certain chemicals and gasses, such as ethylene, toescape and limit the effect of such gasses on the produce.

According to the various embodiments, it is contemplated that the objectrecognition system 672 can be implemented throughout the entirehousehold to monitor the current inventory manifest 560 of products inthe kitchen, bathroom, laundry room, utility rooms, and other portionsof the household to manage consumable and depletable items within thehousehold. The system provides for the efficient management of thecurrent inventory manifest 560, as well as an efficient system forconducting shopping excursions for replacement of consumable anddepletable items that constitute the temporarily stored items 542monitored through the object recognition system 672. In thesenon-limiting locations, the scanning apparatus 544 can be positioned ona wall, door, shelf 640, ceiling, floor, hanging location, or othersimilar location. The scanning apparatus 544 can also be configured tobe stationary, operable with a door 526, operable within or upon asurface, combinations thereof and configured for other similar operativepositions.

While it is contemplated that the scanning apparatus 544 be disposedwithin the refrigerating appliance 510, it is also contemplated that thescanning apparatus 544 can be implemented within various appliance andproduct storage spaces throughout the household. Accordingly, it iscontemplated that various scanning apparatus 544 can be disposed withincabinets, cupboards, pantries, refrigerating appliances 510, laundryrooms, bathroom cabinets, linen closets, storerooms, utility spaces, andother similar places where consumable and depletable temporarily storeditems 542 may be kept. The object recognition system 672 can aggregateall of the information obtained during the various data capturing phases550 of the one or more scanning apparatuses 544 disposed throughout thehouse and aggregate a single current inventory manifest 560 of thehousehold. The current inventory manifest 560 for the household can beitemized and categorized as needed by the user to reveal the status ofvarious temporarily stored items 542 throughout the household.Statistics such as date purchased, date stored, number of times used,number of times moved, expiration date, lifespan, remaining quantity,previous purchases of a particular item, retail price at variousretailers, effective price per unit at various retailers, and othersimilar statistics regarding the various temporarily stored items 542within the current inventory manifest 560. Additionally, the currentinventory manifest 560 can maintain data about space available among thevarious storage surfaces 540 of the various appliances 510 or otherstorage areas of the household.

According to the various embodiments, the various retail stores 710discussed herein can include local markets having physical locationsnear the user's address, such as Walmart, Costco, Sam's Club, Kroger,Home Depot, farmer's markets, “mom and pop” retailers, hardware stores,and the like. Retail stores 710 can also include internet marketingsites (i.e., Amazon.com, ebay.com, Plated™, Blue Apron™, etc.) wherevarious temporarily stored items 542 can be purchased and delivered tothe household from local or remote locations. Retail stores 710 can alsoinclude catalog-type services where items can be purchased via internetor over the phone.

It is contemplated that the object recognition system 672 can provide aproactive system that allows the user to know when items need to bereplaced before the user runs out or before a particular temporarilystored item 542 reaches its expiration date.

Additionally, the object recognition system 672 can include adaptivelearning functions that allow the user's preferences to be incorporatedinto the object recognition system 672. Such preferences and adaptivelearning information can also be used in the formation of shopping lists712, recommendations, comprehensive menu profiles 740 and other aspectsof the object recognition system 672.

It is contemplated that the object recognition system 672 including thecurrent inventory manifest 560, shopping list 712, recipes and otheraspects of the object recognition system 672 can be accessed by the userthrough a portable computing device 630, such as, but not limited to,cell phone, smart phone, wearable computing device, laptop, or otherportable or non-portable computing device. Data 556, 558 collected,manipulated, and presented for use by the user can be stored either onthe refrigerating appliance 510, on the user's computing device, or at athird location such as within a “cloud” storage apparatus. It is alsocontemplated that the user's portable computing device 630 can uploadvarious programs or apps that allow the information provided by thecurrent inventory manifest 560 and various shopping lists 712 to beintegrated with apps provided by various retailers, shopping services,and other services related to the purchase of temporarily stored items542 needed by the user. Many of these apps and programs are currently inuse and may be downloaded upon a user's portable computing device 630.Future apps may be formulated for more efficient use and integration ofthe retailer's programs and apps in conjunction with apps and programsrelated to the object recognition system 672.

Reference is now made to FIGS. 24-34. For clarity and illustrativepurposes, the reference numbers included below relate to FIGS. 24-34.

Referring to FIGS. 24-34, an appliance 1020 includes a temperaturecontrol system 1024. The appliance 1020 may be additionally equippedwith a light source 1028 and a spectrometer 1032. In such embodiments,the spectrometer 1032 works in conjunction with the light source 1028 toobtain a measured value of a foodstuff 1036. The measured value is atleast one of an absorbance value, a transmittance value, and areflectance value. In various embodiments, the measured value from thefoodstuff 1036 may be compared to a reference value for the foodstuff1036. In some embodiments, the temperature control system 1024 mayprovide either controlled heat or controlled cold to a region 1040 ofthe appliance 1020. For example, the region 1040 of the appliance 1020may be an interior cavity 1044 of a refrigerator (FIG. 24). The interiorcavity 1044 of the refrigerator may be equipped with one or more storagecompartments 1048. Alternatively, the region 1040 of the appliance 1020may be a cook surface 1052 of a stove or cooktop (FIGS. 30 and 31). Thecook surface 1052 may be equipped with one or more cooking areas 1056.The region 1040 of the appliance 1020 may also be a cooking cavitywithin an oven, microwave oven, toaster oven, toaster, or the like. Itis contemplated that the temperature control system 1024 may beconfigured to provide controlled head and controlled cold to the region1040 of the appliance 1020. In one such example, the refrigerator may beprovided with controlled heat to accelerate food ripeness, freshness, orto generally accelerate the transition of the foodstuff 1036 to adesired state. In another example, a freezer may be provided with apre-thaw section that provides controlled heat solely to the pre-thawsection; and the pre-thaw section may be selectively controlled for whento have the foodstuff 1036 at least partially thawed. In examples wherethe appliance 1020 is a cooking appliance, the region 1040 may beprovided with controlled heat and controlled cold selectively, forexample, to at least partially cool the foodstuff 1036 to a servingtemperature.

Referring to the embodiment illustrated in FIG. 24, the appliance 1020may be a refrigerator, freezer, or a combination thereof. Accordingly,the region 1040 of the appliance 1020 is provided with the controlledcold environment by the temperature control system 1024. The appliance1020 is additionally equipped with the light source 1028. The appliance1020 may further include at least one of the spectrometer 1032 (FIGS.26A and 26B), an imager 1060, and a gas detector 1064 (FIGS. 25A-26B).The spectrometer 1032 and the light source 1028 are utilized togethersuch that the spectrometer 1032 and the light source 1028 work inconjunction to obtain the measured value of the foodstuff 1036 (FIGS.25A-26B). In embodiments that employ the spectrometer 1032, the measuredvalue may be at least one of an absorbance value, a transmittance value,and a reflectance value. Once the measured value has been obtained bythe spectrometer 1032, then the measured value may be compared to areference value for the foodstuff 1036. The reference value for thefoodstuff 1036 may be stored in a database. The foodstuff 1036 may bestored, for example, in one of the storage compartments 1048. In someembodiments, each of the one or more storage compartments 1048 may beequipped with an associated spectrometer 1032. In the depictedembodiment, the appliance 1020 is provided with a refrigerated section1068, a frozen section 1072, a plurality of access doors 1076, and aplurality of storage shelves 1080. As disclosed herein, features thatpertain to the storage compartments 1048 may apply in part or in full tothe storage shelves 1080. Similarly, features that pertain to thestorage shelves 1080 may apply in part or in full to the storagecompartments 1048.

Referring now to FIGS. 25A and 25B, the storage compartment 1048 isequipped with a storage area 1084 that is defined by a plurality ofvertical walls 1088 and a horizontal bottom 1092. One of the verticalwalls 1088 is equipped with a handle 1096 such that a user may actuatethe storage compartment 1048 from a closed position to an open position.One or more of the storage compartments 1048 may be equipped with atleast one of the light source 1028, the spectrometer 1032 (FIGS. 26A and26B), the imager 1060, and the gas detector 1064. For example, in thedepicted embodiment the storage compartment 1048 is equipped with theimager 1060 and the gas detector 1064. The imager 1060 and/or the gasdetector 1064 may be configured to monitor at least one of the storagecompartments 1048. The imager 1060 may be utilized for various purposeswithin the appliance 1020 (FIG. 24). For example, the imager 1060 can beutilized to determine a quantity and/or a type of the foodstuff 1036. Ingeneral, the imager 1060 is configured to monitor the storagecompartments 1048 and provide data that relates to the quantity and/orthe type of the foodstuff 1036. The gas detector 1064 may be configuredto provide data relating to a state of the foodstuff 1036 (e.g., not yetripe, will be ripe soon, ripe, will spoil soon, spoiled, etc.) The stateof the foodstuff 1036 may be determined by detecting and/or analyzinggaseous components 1100 that are given off by the foodstuff 1036 (e.g.,volatile organic compounds). The data provided by the imager 1060 and/orthe gas detector 1064 may be communicated to a controller 1104 that iscapable of analyzing the data provided. For example, the controller 1104may analyze the data provided by referencing a database. In variousembodiments, the gas detector 1064 may be provided with a plurality ofchannels (e.g., 4, 8, 16, etc.). The gas detector 1064 may be configuredas a device with a single channel per device or as a device withmultiple channels per device. Regardless of the number of channelsprovided in the gas detector 1064, the gas detector 1064 is configuredto collect data on multiple items simultaneously and from a distance ofless than about 5 cm, greater than about 5 cm, greater than about 10 cm,greater than about 15 cm, greater than about 20 cm, and/or combinationsor ranges thereof. In some embodiments, the gas detector 1064 mayoperate on the principles of gas chromatography. Alternatively, the gasdetector 1064 may be a gas chromatography instrument configured forcoupling to the appliance 1020.

Referring to FIGS. 26A and 26B, the storage compartment 1048 includesthe spectrometer 1032, the imager 1060, and the gas detector 1064. Thespectrometer 1032 may be an independent element from the imager 1060 andthe gas detector 1064. Alternatively, the spectrometer 1032 may beintegrated with at least one of the imager 1060 and the gas detector1064. The spectrometer 1032 may utilize various forms of electromagneticradiation and/or mechanical waves to detect various properties of thefoodstuff 1036. For example, the electromagnetic radiation may beinfrared light, visible light, ultraviolet light, or combinationsthereof. The mechanical waves may be acoustic waves, such as radar orultrasound, in various examples. Data provided by the spectrometer 1032may include information about a color, shape, or other identifyinginformation that aids in the identification of the type and/or quantityof the foodstuff 1036.

Referring again to FIGS. 26A and 26B, the data collected by thespectrometer 1032, the imager 1060, and/or the gas detector 1064 may becommunicated to the controller 1104. The controller 1104 thenaccumulates the data collected and can analyze and/or compare the datato reference values to determine information about the foodstuff 1036,for example, type, quantity, ripeness, etc. While the light source 1028,the spectrometer 1032, the imager 1060, and the gas detector 1064 aredepicted as being operably coupled to one of the vertical walls 1088,the present disclosure is not so limited. Accordingly, the light source1028, the spectrometer 1032, the imager 1060, and the gas detector 1064may be alternatively positioned, such as on an underside of one of thestorage shelves 1080 (FIG. 24) that corresponds to the associatedstorage compartment 1048. Similarly, the controller 1104 may bealternatively positioned within the appliance 1020. For example, thecontroller 1104 may be positioned outside of the interior cavity 1044(FIG. 24) at a location that is remote from the various sensorspositioned in the interior cavity 1044.

Referring now to FIGS. 27A and 27B, the appliance 1020 includes theimager 1060, the gas detector 1064, and a weight scale 1108. In thedepicted embodiment, the weight scale 1108 is positioned on thehorizontal bottom 1092 of the storage compartment 1048. It iscontemplated that the weight scale 1108 may be alternatively oradditionally positioned on one or more of the vertical walls 1088.Positioning the weight scale 1108 or additional weight scales 1108 onone or more of the vertical walls 1088 may be beneficial as the weightof the foodstuff(s) 1036 that are stored in the storage compartments1048 may be at least partially supported on one or more of the verticalwalls 1088. For example, large items resting on the horizontal bottom1092 and leaning against one or more of the vertical walls 1088. In someembodiments, the appliance may further include the spectrometer 1032(FIGS. 26A and 26B). The controller 1104 (FIGS. 26A and 26B) may beemployed to receive data from at least one of the spectrometer 1032, theimager 1060, the gas detector 1064, and the weight scale 1108. The datamay be utilized to identify a type and/or quantity of the foodstuff1036. The data may be additionally or alternatively utilized todetermine a used-to-free-space ratio of at least one of the storagecompartments 1048. The used-to-free-space ratio may be utilized toaccount for variations in an available volume of the storage compartment1048. The used-to-free-space ratio can help distinguish whether there isa small quantity of foodstuffs 1036 present that are spoiled (e.g., twostrawberries) or if there is a large quantity of foodstuffs 1036 thathave reached their optimal ripeness (e.g., thirty strawberries).

Referring to FIG. 28, the appliance 1020 is equipped with the imager1060 and an array of sensors 1112. The array of sensors 1112 includes afirst sensor 1116, a second sensor 1120, a third sensor 1124, and atemperature sensor 1128. One or more of the first sensor 1116, thesecond sensor 1120, the third sensor 1124, and the temperature sensor1128 may be provided as an integrated unit, such as that shown in thedepicted embodiment for the first, second, and third sensors 1116, 1120,1124. Additionally or alternatively, one or more of the first sensor1116, the second sensor 1120, the third sensor 1124, and the temperaturesensor 1128 may be individual or separate sensors, such as thetemperature sensor 1128 in the depicted embodiment. One of the sensorsin the array of sensors 1112 may be the gas detector 1064 that isconfigured to detect the gaseous components 1100 of the foodstuff(s)1036. The remaining sensors in the array of sensors 1112 may be at leastone of the spectrometer 1032, the imager 1060, or another sensor that isconfigured to provide information about the foodstuff 1036 that aids indetermining a type, quantity, and/or ripeness of the foodstuff 1036.Additionally or alternatively, the array of sensors 1112 may provideinformation about a degree of doneness of the foodstuff 1036. Forexample, the array of sensors 1112 may be configured to provide datathat can be utilized to inform a user when a cheesecake has fully set orwhen a gelatinous dish has fully set. The array of sensors 1112 may alsobe utilized to extrapolate when the foodstuff 1036, such as thecheesecake, will be fully set and ready to enjoy.

Referring again to FIG. 28, the controller 1104 (FIGS. 26A and 26B) maycompare a first state of the foodstuff 1036 to a second state of thefoodstuff 1036. In the present embodiment, where the temperature controlsystem 1024 (FIG. 24) provides a cold environment, the first state ofthe foodstuff 1036 may be a preserved state and the second state of thefoodstuff 1036 may be a not fully preserved state. Examples of the notfully preserved state may include, but are not limited to, the foodstuff1036 being past peak ripeness, the foodstuff 1036 being outside of apreferred freshness “window,” the foodstuff 1036 being within a finalripeness window, the foodstuff 1036 reaching a spoiled state, and thelike. Monitoring the transition of the foodstuff 1036 from the preservedstate to the spoiled state may be accomplished, for example, by at leastone of the spectrometer 1032, the imager 1060, the gas detector 1064,and/or the weight scale 1108 (FIGS. 27A and 27B). For example, thefoodstuff 1036 may be monitored by the gas detector 1064 to detect achange in the gaseous components 1100 that are given off by thefoodstuff 1036 as the foodstuff 1036 transitions from the preservedstate to one of a ripened state and the spoiled state. The controller1104 may utilize data collected from the gas detector 1064 and at leastone of the spectrometer 1032, the imager 1060, and the weight scale 1108to determine a preferred storage temperature, humidity, and/or airvelocity for at least one of the storage compartments 1048. The weightscale 1108 may provide data relating to the freshness of the foodstuff1036. For example, the weight scale 1108 may register a decrease in aweight of the foodstuff 1036 as the foodstuff 1036 loses moisture due tothe transition of the foodstuff 1036 to a not fully preserved state(e.g., a spoiled state).

Referring now to FIGS. 29A and 29B, the data provided by thespectrometer 1032, the imager 1060, the gas detector 1064, the weightscale 1108, the array of sensors 1112, the first sensor 1116, the secondsensor 1120, the third sensor 1124, the temperature sensor 1128, anyother sensors that are employed, and/or combinations thereof can beutilized by the controller 1104 to determine a current state of thefoodstuff 1036 that may be updated or logged periodically (e.g., hourly,daily, weekly, etc.). For example, the data provided may be consolidatedinto a plot, graph, or table such as that shown in FIG. 29A. In theexample shown in FIG. 29A, the data provided for Day 1 and Day 2 havebeen consolidated and are shown in solid lines. See the precedingfigures and their associated descriptions for depiction of the varioussensors and components of the appliance 1020. Using the data collectedon Day 1 and Day 2, the controller 1104 may then extrapolate a futurestate of the foodstuff 1036, as shown in Day 3, Day 4, and Day 5 of FIG.29A, which are shown in broken lines. The extrapolated data may beupdated on the same schedule as the data being logged about thefoodstuff 1036 such that the extrapolated data can be adjusted to betterfit the progression of the foodstuff 1036. The extrapolated data can beused to estimate, for example, when the foodstuff 1036 will be ripe,when the foodstuff 1036 will spoil, when the foodstuff 1036 will befully set, and/or when the foodstuff 1036 will be ready to serve.

Referring again to FIGS. 29A and 29B, in addition to extrapolatinginformation about the foodstuff 1036 by referencing the database, it iscontemplated that the data collected from the sensors (e.g.,spectrometer 1032, gas detector 1064, array of sensors 1112, etc.) maybe utilized to further develop the database or be stored and accumulatedas a user-specific database. Said another way, the database may be a“living” database that is constantly improving its predictive accuracyregarding the foodstuff 1036 by updating itself with the data that iscollected from the appliances 1020 that are installed in homes aroundthe world. Additionally or alternatively, the data collected fromindividual appliances 1020 may be stored as individual databases thatare specific and customized to the particular appliance 1020 from whichthe data was originally collected. For example, the data collected overa five day period about the foodstuff 1036, such as that depicted inFIG. 29B, may be referenced against the larger worldwide database todetermine any deviations from the progression of the foodstuff 1036relative to the average progression of the foodstuff 1036 as determinedby the multitude of appliance 1020 utilized around the world. Anydeviations from the progression of the foodstuff 1036 can then be storedas the individual and customized database that is specific to theparticular appliance 1020. Accordingly, an individual database may beaccumulated for the individual appliances 1020 to account for variationsthat may exist while maintaining the ability to reference the largerworldwide or regional database. It may be beneficial to developregionally specific databases with the data collected from theappliances 1020 as growing, harvesting, and/or transportation practicesare likely to vary based on factors such as time of year and destinationof the foodstuff 1036. In various examples, the appliance 1020 mayutilize a variety of diagnostic inputs from the sensors as a way to logvariances in food freshness maturity. Said another way, the diagnosticinputs can be utilized to help track when the foodstuff 1036 has reacheda mature or ripened state at a point in time that falls outside of apredicted or average timeframe or time window.

Referring now to FIGS. 30 and 31, in some embodiments, the temperaturecontrol system 1024 provides controlled heat to the region 1040 of theappliance 1020, such as with the cooking areas 1056 of a stove orcooktop. In such embodiments, the light source 1028, the spectrometer1032, the imager 1060, the gas detector 1064, the access door 1076, thecontroller 1104, the weight scale 1108 (FIGS. 27A and 27B), the array ofsensors 1112, the first sensor 1116, the second sensor 1120, the thirdsensor 1124, and/or the temperature sensor 1128 may be employed invarious combinations. Two examples of such combinations orconfigurations are shown in FIGS. 30 and 31; however, the presentdisclosure is not so limited. Regardless of the number of sensorsemployed, the sensors are generally configured to observe one or moremonitored areas, such as the cooking area 1056 of the appliance 1020.The gas detector 1064 may be coupled to and/or in communication with atleast one of the spectrometer 1032 and the imager 1060 such that themonitored area(s) are monitored to identify the first state of thefoodstuff 1036. Once the first state of the foodstuff 1036 has beenidentified, the controller 1104 may compare the first state of thefoodstuff 1036 with the second state of the foodstuff 1036. The firststate and the second state may be identified by the controller 1104comparing the data collected by the sensor(s) with the database thatcontains information about the foodstuff 1036, as collected by the sameor similar sensors to those employed in the appliance 1020. In thepresent embodiments, the temperature control system 1024 is configuredto provide heat to the one or more monitored areas such that thefoodstuff 1036 can be cooked (e.g., in a pan 1132). Accordingly, thefirst state of the foodstuff 1036 may be an uncooked state and thesecond state of the foodstuff 1036 may be a cooked state or a desiredcooked state (e.g., rare, medium rare, medium, medium well, or welldone). In some embodiments, the gas detector 1064 may detect a change inthe gaseous components 1100 that are given off by the foodstuff 1036 asthe foodstuff transitions from the uncooked state to the cooked state.While the imager 1060, the gas detector 1064, and the array of sensors1112 are depicted as coupled to a support surface, such as a wall 1136that is proximate the appliance 1020, it is contemplated that thesensors employed may be alternatively positioned. For example, thesensors that are utilized in a particular embodiment may be positionedin a range or hood that is positioned vertically above or below theappliance 1020. Alternatively, the sensors may be positioned about aperimeter 1140 of the cooking area(s) 1056 or along a control tower 1144of the appliance 1020. Further, the sensors may be integrated into atemperature probe (e.g., meat probe) or into a cooking dish (pan, dish,cookie sheet, etc.). In some embodiments, a ventilation system may beemployed that guides air toward the gas detector 1064 to aid in datacollection. The ventilation system may be employed in either thecontrolled cold or controlled heat examples disclosed herein. While thepreceding description is directed to the region 1040 being the cookingareas 1056 of the stove or cooktop, it is contemplated that theseconcepts may similarly apply to a cooking cavity (e.g., oven cavity,microwave cavity, etc.) without departing from the present disclosure.

Referring to FIG. 32, a schematic representation of the appliance 1020is shown, according to one embodiment of the refrigerator and/orfreezer. The controller 1104, such as an appliance control unit, isconfigured to monitor a plurality of areas within the interior cavity1044 (FIG. 24) of the appliance 1020. The monitored area, such as thestorage shelves 1080 or storage compartments 1048 may include, but arenot limited to, a first storage shelf 1148, a second storage shelf 1152,a third storage shelf 1156, a crisper drawer 1160, an access door shelf1164, a cheese drawer 1168, and a meat drawer 1172. Each of themonitored areas is equipped with one or more of the sensors disclosedherein, such as the gas detector 1064. It is contemplated that a singlesensor may be configured to monitor more than one of the monitored areaswithout departing from the concepts disclosed herein. Additionally,sensors not expressly stated in this disclosure are considered withinthe scope of the disclosure; however, for the sake of brevity anexhaustive list has not been recited. The controller 1104 is coupled tothe sensors such that communication is enabled between the sensors andthe controller 1104. The data collected by the sensors can becommunicated to at least one of the controller 1104 and a Wi-Fi module1176. The Wi-Fi module 1176 may be configured to transmit the datacollected to a user and/or the database. The data collected may then bestored, additionally processed, and/or analyzed. Additionally, the Wi-Fimodule 1176 may utilize data collected from at least one of the sensors(e.g., the imager 1060) to search the internet, an internal database,and/or the Cloud in an effort to identify the foodstuff 1036 (e.g.,visually and/or optically).

Referring now to FIG. 33, a method 1200 of operating an applianceincludes step 1204 of initiating a procedure for operating the appliance1020. Once the procedure for operating the appliance has been initiated,the method 1200 advances to the step 1208 of determining if one of thestorage compartments 1048 is in an open position. If step 1208determines that one of the storage compartments 1048 is in the openposition, then the method 1200 advances to step 1212 of preventing powerfrom being supplied to the sensors associated with the open storagecompartment(s) 1048. If step 1208 determines that none of the storagecompartments 1048 is in the open position, then the method 1200 advancesto step 1216 of supplying power to the sensors of the storagecompartments 1048. Once power has been supplied to the sensors in step1216, the method 1200 advances to step 1220 of determining if thestorage compartment 1048 has been opened since the last time thesensor(s) have measured the storage compartment 1048. If step 1220determines that the storage compartment 1048 has not been opened sincethe last time the sensor(s) have measured the storage compartment 1048,then the method 1200 advances to step 1224 of measuring the storagecompartment 1048 with the sensors (e.g., measure weight and gaseouscomponent 1100 concentrations). If step 1220 determines that the storagecompartment 1048 has been opened since the last time the sensor(s) havemeasured the storage compartment 1048, then the method 1200 advances tostep 1228 of monitoring the storage compartment 1048 with the sensors(e.g., monitoring weight and gaseous component 1100 concentrations).Simultaneous to or sequential to the performance of step 1228, themethod 1200 performs step 1232 of communicating a current status of thecontents of the storage compartment 1048 (e.g., the foodstuff 1036) toat least one of the controller 1104 and the Wi-Fi module 1176. Followingstep 1228 and/or step 1232, the method 1200 advances to step 1236 ofdetermining if any of the sensors are registering a measurement thatfalls outside of a predetermined threshold. If step 1236 determines thatnone of the sensors has registered a measurement that falls outside ofthe predetermined threshold, then the method 1200 returns to step 1228.If step 1236 determines that at least one of the sensors have registereda measurement that falls outside of the predetermined threshold, thenmethod 1200 advances to step 1240 of communicating a notification to auser that corresponds to the reading that fell outside of thepredetermined threshold. This notification may be transmitted to theWiFi module 1176, a control module, the Cloud, to the users mobiledevice, displayed on an exterior of the appliance 1020 (e.g.,door-mounted screen), and/or combinations thereof. For example, the usermay be notified that one or more of the foodstuffs 1036 stored in aparticular storage compartment 1048 have ripened, fully set, or spoiled.

Referring to FIG. 34, a method 1300 of assembling an appliance includesstep 1304 of initiating assembly of the appliance 1020. Once assembly ofthe appliance 1020 has been initiated, the method 1300 advances to step1308 of coupling the temperature control system 1024 to the appliance1020. Next, the method 1300 advances to step 1312 of coupling the gasdetector 1064 to the appliance 1020. Then, the method 1300 advances tothe step 1316 of coupling the imager 1060 to the appliance 1020. Atleast one of the imager 1060 and the gas detector 1064 are coupled tothe controller 1104 in step 1320. Next, the method 1300 advances to step1324 of configuring the gas detector 1064 to monitor an area of theappliance 1020 (e.g., the storage compartment(s) 1048, the cooking areas1056, etc.). Then, the method 1300 advances to step 1328 of configuringthe controller 1104 to collect data from at least one of the imager 1060and the gas detector 1064. The method 1300 may further include step 1332of coupling the spectrometer 1032 to the appliance 1020. In someembodiments, the method 1300 may also include step 1336 of providing thelight source 1028 that works in conjunction with at least one of theimager 1060 and the spectrometer 1032. Further, the method 1300 mayinclude step 1340 of configuring the controller 1104 to provideinformation about the foodstuff 1036 in the area of the appliance 1020that is being monitored.

In the various embodiments disclosed herein, the sensors may be shownand described as rigid elements. However, the present disclosure is notso limited. For example, it is contemplated that one or more of thesensors utilized may be movable, such as removably coupled to theappliance 1020, capable of pivotable motion relative to the appliance1020, capable of rotatable motion relative to the appliance 1020,capable of being extended away from a coupling surface of the appliance1020 (e.g., retractable), and combinations thereof.

Referring now to FIG. 35, a method 1400 of operating an applianceincludes step 1404 of providing power to the appliance. Once power hasbeen provided to the appliance, the method 1400 may advance to step 1408of sensing a compartment access event. The term compartment as usedherein is not intended to limit the disclosure to drawers or the like.Rather, the term compartment is intended to refer to any space or areain which content, such as the foodstuff 1036, is stored. Accordingly,the compartment access event is intended to refer to sensing whether auser has interacted with the appliance in a physical manner (e.g.,opening an access door, moving or removing an item stored, etc.). Uponsensing a compartment access event, the method 1400 may advance to step1412 of updating a compartment access state (e.g., recently accessed,not recently accessed, etc.). Next, the method 1400 may advance to step1416 of determining an access door position (e.g., open or not open). Ifthe access door position is determined to be open in step 1416, then themethod 1400 may return to step 1408 of sensing a compartment accessevent and proceed as described above. However, if the access doorposition is determined to be not open or closed, then the method 1400may advance to step 1420 of measuring the content (e.g., the foodstuff1036) that is stored in the compartment. Once the content stored in thecompartment has been measured, the method 1400 may advance to step 1424of communicating the data measured or collected. The data communicatedin step 1424 may be communicated, for example, to the database, the WiFimodule, the Cloud, and/or generally communicated as a notification to auser. Next, method 1400 may advance to step 1428 of monitoring thecompartment(s). The monitoring of the compartment(s) may be a non-eventdriven food status step. Said another way, the monitoring of thecompartment(s) in step 1428 is executed independent of whether a userhas interacted with the appliance. As the compartment(s) are beingmonitored at step 1428, the method 1400 may advance to either step 1430or step 1432. Step 1430 involves updating the compartment content dataand returning to step 1416 in method 1400, where the method 1400proceeds as described above. Step 1428 may advance to step 1430, forexample, when a change or transition has been detected for the contentof the compartment (e.g., preserved state to not fully preserved state).The status of the contents in the compartment(s) may be utilized as aweighted input for determining an interval of monitoring the contentsbetween compartment access events. The status of the contents may bedetermined by at least one of, but is not limited to, weight, quantity,type, maturity, freshness, etc. Step 1432 involves communicating thedata measured or collected during the monitoring of step 1428. The datacommunicated in step 1432 may be communicated, for example, to thedatabase, the WiFi module, the Cloud, and/or generally communicated as anotification to a user. In method 1400, if the compartment access statuscannot be determined (e.g., for shelves), then the method 1400 maydefault to assuming the compartment was not accessed or was accessedbased on a user setting or a particular application (e.g., heated orcooled examples). Similarly, if the method 1400 cannot determine anaccess door position, then the method 1400 may default to assuming theaccess door is not open or open based on a user setting or a particularapplication (e.g., heated or cooled examples).

It will be understood by one having ordinary skill in the art thatconstruction of the described device and other components is not limitedto any specific material. Other exemplary embodiments of the devicedisclosed herein may be formed from a wide variety of materials, unlessdescribed otherwise herein.

For purposes of this disclosure, the term “coupled” (in all of itsforms, couple, coupling, coupled, etc.) generally means the joining oftwo components (electrical or mechanical) directly or indirectly to oneanother. Such joining may be stationary in nature or movable in nature.Such joining may be achieved with the two components (electrical ormechanical) and any additional intermediate members being integrallyformed as a single unitary body with one another or with the twocomponents. Such joining may be permanent in nature or may be removableor releasable in nature unless otherwise stated.

It is also important to note that the construction and arrangement ofthe elements of the device as shown in the exemplary embodiments isillustrative only. Although only a few embodiments of the presentinnovations have been described in detail in this disclosure, thoseskilled in the art who review this disclosure will readily appreciatethat many modifications are possible (e.g., variations in sizes,dimensions, structures, shapes and proportions of the various elements,values of parameters, mounting arrangements, use of materials, colors,orientations, etc.) without materially departing from the novelteachings and advantages of the subject matter recited. For example,elements shown as integrally formed may be constructed of multiple partsor elements shown as multiple parts may be integrally formed, theoperation of the interfaces may be reversed or otherwise varied, thelength or width of the structures and/or members or connector or otherelements of the system may be varied, the nature or number of adjustmentpositions provided between the elements may be varied. It should benoted that the elements and/or assemblies of the system may beconstructed from any of a wide variety of materials that providesufficient strength or durability, in any of a wide variety of colors,textures, and combinations. Accordingly, all such modifications areintended to be included within the scope of the present innovations.Other substitutions, modifications, changes, and omissions may be madein the design, operating conditions, and arrangement of the desired andother exemplary embodiments without departing from the spirit of thepresent innovations.

It will be understood that any described processes or steps withindescribed processes may be combined with other disclosed processes orsteps to form structures within the scope of the present device. Theexemplary structures and processes disclosed herein are for illustrativepurposes and are not to be construed as limiting.

It is also to be understood that variations and modifications can bemade on the aforementioned structures and methods without departing fromthe concepts of the present device, and further it is to be understoodthat such concepts are intended to be covered by the following claimsunless these claims by their language expressly state otherwise.

The above description is considered that of the illustrated embodimentsonly. Modifications of the device will occur to those skilled in the artand to those who make or use the device. Therefore, it is understoodthat the embodiments shown in the drawings and described above is merelyfor illustrative purposes and not intended to limit the scope of thedevice, which is defined by the following claims as interpretedaccording to the principles of patent law, including the Doctrine ofEquivalents.

What is claimed is:
 1. An appliance, comprising: a temperature controlsystem; a light source; and a spectrometer that works in conjunctionwith the light source to obtain a measured value of a foodstuff, whereinthe measured value is at least one of an absorbance value, atransmittance value, and a reflectance value, and wherein the measuredvalue from the foodstuff is compared to a reference value for thefoodstuff.
 2. The appliance of claim 1, further comprising: one or morestorage compartments that are monitored by the spectrometer.
 3. Theappliance of claim 2, wherein each of the one or more storagecompartments is equipped with the spectrometer.
 4. The appliance ofclaim 2, further comprising: an imager that is configured to monitor atleast one of the one or more storage compartments.
 5. The appliance ofclaim 2, further comprising: a gas detector that is configured tomonitor at least one of the one or more storage compartments.
 6. Theappliance of claim 2, further comprising: an imager that is configuredto monitor at least one of the one or more storage compartments; and agas detector that is configured to monitor at least one of the one ormore storage compartments.
 7. The appliance of claim 6, furthercomprising: a controller that receives data from at least one of thespectrometer, the imager, and the gas detector, wherein the data isutilized to identify at least one of a type and a quantity of thefoodstuff.
 8. An appliance, comprising: a temperature control system; alight source; a controller; one or more monitored areas; and a gasdetector coupled, and in communication, with at least one of an imager,a weight scale, and a spectrometer such that the one or more monitoredareas are monitored to identify a first state of a foodstuff, whereinthe controller compares the first state of the foodstuff with a secondstate of the foodstuff.
 9. The appliance of claim 8, wherein thetemperature control system is configured to provide heat to the one ormore monitored areas.
 10. The appliance of claim 9, wherein the firststate of the foodstuff is an uncooked state of the foodstuff, andwherein the second state of the foodstuff is a cooked state of thefoodstuff.
 11. The appliance of claim 10, wherein the gas detectordetects a change in gases that are given off by the foodstuff as thefoodstuff transitions from the uncooked state to the cooked state. 12.The appliance of claim 8, wherein the temperature control system isconfigured to provide a cold environment to the one or more monitoredareas such that the foodstuff may be preserved.
 13. The appliance ofclaim 12, wherein the first state of the foodstuff is a preserved stateof the foodstuff, and wherein the second state of the foodstuff is a notfully preserved state of the foodstuff.
 14. The appliance of claim 13,wherein the gas detector detects a change in gases that are given off bythe foodstuff as the foodstuff transitions from the preserved to stateto one of a ripened state and the spoiled state.
 15. The appliance ofclaim 12, wherein the controller utilizes data collected from the gasdetector and at least one of the imager and the spectrometer todetermine a preferred storage temperature for at least one of the one ormore monitored areas.
 16. The appliance of claim 12, wherein the one ormore monitored areas are one or more storage compartments in arefrigerator.
 17. A method of assembling an appliance, comprising thesteps of: coupling a temperature control system to the appliance;coupling a gas detector to the appliance; coupling an imager to theappliance; coupling at least one of the gas detector and the imager to acontroller; configuring at least one of the gas detector and the imagerto monitor an area of the appliance; and configuring the controller tocollect data from at least one of the gas detector and the imager. 18.The method of assembling an appliance of claim 17, further comprisingthe step of: coupling a spectrometer to the appliance.
 19. The method ofassembling an appliance of claim 18, further comprising the step of:providing a light source that works in conjunction with at least one ofthe imager and the spectrometer.
 20. The method of assembling anappliance of claim 17, further comprising the step of: configuring thecontroller to provide information about a foodstuff in the area of theappliance that is monitored.